How to split a string variable into an array of hyphenated string variables of certain width

Question

I have a variable (defined with \csdef{id}{value}) which contains a normal (albeit long) string of text.

I need a macro (?) which does roughly the following:

• Squeezes this string into a "virtual" (invisible) minipage
• To do this, it uses standard hyphenation rules to break the string into separate lines
• This minipage has a certain width, which I've specified
• The lines are written, say, into a string array (maybe with the same \csdef{id}{value} command).

These two questions are about something very similar, but I don't have some specially structured text which I need to parse. I just need to make use of the standard hyphenation function of LaTeX, except "behind the scenes", without actually typesetting anything.

Answer 1

With Luatex you can recreate character tokens from typeset box nodes, a direction not possible in classic tex.

\documentclass{article}
\usepackage{color}
\begin{document}

\def\dohyphens#1#2{%
\directlua{
res='\string\\gdef\string\\#1\iftrue\string{\else}\fi'
local G = node.id("glyph")
local D = node.id("disc")
local K = node.id("kern")
gethyph = function (head)
local l = -1
for n in node.traverse(head) do
 if n.id==G then
   if l\string~=G then
     res = res ..'\string\\HYchars\iftrue\string{\else}\fi'
   end
     res = res .. string.char(n.char)
 else 
   if l\string==G then
     res = res .. '\iffalse{\else\string}\fi'
   end
 if n.id==D then
   res = res .. '\string\\HYhyphen'
 else if (n.id ==K) then
 else if l==G or l==D then
    res = res .. '\string\\HYspace'
 end
 end
 end
 end
 l=n.id
end
res = res ..'\iffalse{\else\string}\fi'
return true
end 
luatexbase.add_to_callback('pre_linebreak_filter',gethyph,'get hyphens')
}%
\setbox0\vbox{\hsize\maxdimen\ttfamily\hyphenchar\font=`\-#2}%
\directlua{
luatexbase.remove_from_callback('pre_linebreak_filter','get hyphens')
tex.sprint(res)
}%
}

\protected\def\HYchars#1{\color{red}{#1}}
\protected\def\HYhyphen{\colorbox{yellow}{\color{blue}{---}}}
\protected\def\HYspace{ }

\dohyphens{tmp}{supercalifragilisticexpialidocious}

\typeout{tmp is}
\typeout{\meaning\tmp}

\tmp
\end{document}

The above splits up supercalifragilisticexpialidocious putting the result into \tmp which it shows on the terminal as

tmp is
macro:->\HYchars {su}\HYhyphen \HYchars {p}\HYchars {er}\HYhyphen \HYchars {cal
}\HYhyphen \HYchars {ifrag}\HYhyphen \HYchars {ilis}\HYhyphen \HYchars {tic}\HY
hyphen \HYchars {ex}\HYhyphen \HYchars {pi}\HYhyphen \HYchars {ali}\HYhyphen \H
Ychars {do}\HYhyphen \HYchars {cious}\HYspace 

so basically runs of letters end up in \HYchars discretionary hyphens as \HYhyphen inter-letter kerns are removed (although there should be no kerns as using ttfamily to also avoid ligatures) and anything else is \HYspace

You can define the \HY... commands to do whatever you want here it just adds some colour and exaggerated, non discretionary, dashes

---

Following comments in the comments it appears you want something more like this, using the post_linebreak_filter to collect up lines after linebreaking.

\documentclass{article}
\usepackage{color}
\begin{document}

\def\foo{a supercalifragilisticexpialidocious one two three four
supercalifragilisticexpialidocious one two three four
apples oranges carrots bananas
difficult find finger floor Vouch}


\def\showlines{\directlua{tex.print(res)}}

\def\getlines#1#2{%
\directlua{
local N = node.id("hlist")
local G = node.id("glyph")
local D = node.id("disc")
local K = node.id("kern")
local GLUE = node.id("glue")
%
getlines = function (head)
res=''
for vnode in node.traverse(head) do
if vnode.id==N then
  res=res .. getlinetext(vnode.head)
end
end
return true
end
%
getlinetext = function (head)
local linetext=''
for n in node.traverse(head) do
 if n.id==G then
   if n.subtype==2 then
     for nn in node.traverse(n.components) do
      linetext = linetext  .. string.char(nn.char)
     end
   else
     linetext = linetext   .. string.char(n.char)
   end
 else if n.id==GLUE then
     linetext = linetext .. ' '
 else if n.id==D then
     if n.replace \string~= nil then
       for nn in node.traverse(n.replace) do
        if nn.char \string~= nil then
          if nn.char==14 then
            linetext = linetext .. 'ffi' % OT1 encoding specific
          else
            linetext = linetext .. string.char(nn.char)
           end
         end
       end
     end
 end
 end
 end
end
texio.write_nl('\string\\textline{' .. linetext .. '}')
return '\string\\textline{' .. linetext .. '}'
end 
luatexbase.add_to_callback('post_linebreak_filter',getlines,'get lines')
}%
\vbox{\hsize#1\relax#2}% show typeset result to check: add \setbox0= to hide
\directlua{
luatexbase.remove_from_callback('post_linebreak_filter','get lines')
}%
}



\getlines{5cm}{\foo}

produces the following linebreaks

\bigskip

\def\textline#1{\texttt{#1}\par}

\showlines

\end{document}

Answer 2

It is possible to access to the word parts after complete hyphenation has been performed, but only in the form of an already typeset box, which is pretty much what \showhyphens in xltxtra or the macros in the testhyphens package do. However these word parts cannot be retrieved at the token processing stage, because hyphenation happens when paragraphs are being split into lines and there's no user control on that (other than for setting parameters).

One might imagine a two stage pass, where the word parts are written in the log file via a \showhyphens like macro; a filter in Perl or similar language might retrieve the word parts from the log file between LaTeX runs.

Maybe this is possible with LuaTeX, it would be an interesting research project.