All geek questions in one place

Faster text processing in BASH

My question is speed. I have a bash script that parses information from TheTvDb.com. It downloads about 40,000 lines of data, and then reduces to 5,000 lines of data that are written to the hard drive. He then reads the file and analyzes it into several files, which are later used as a lookup table. It basically takes all the information that it sees before each "/ episode", and writes it to a specific file, and then performs a reset for the next.

It should sync in the "/ Episode" tag because there is a "FirstAired" tag outside of the episode tags. This ensures that the data will be drawn sequentially, and then depending on each individual tag related to the episode.

Here is the code in question.

  if [ -f "$mythicalLibrarian/$NewShowName/$NewShowName.xml" ]; then
   Ename=""
   actualEname=""
   FAired=""
   SeasonNr=""
   EpisodeNr=""
    recordNumber=0

    echo "Parsing Downloaded information: $NewShowName.xml "
    while read line
   do

     if [[ $line == \<\/Episode\> ]]; then
      (( ++recordNumber ))
      echo -ne "Building Record:$recordNumber ${actualEname:0:20}            \r" 1>&2 
     echo "$actualEname" >> "$mythicalLibrarian/$NewShowName/$NewShowName.actualEname.txt"&
      Ename=`echo "$actualEname" |sed 's/;.*//'`
     echo "$Ename" >> "$mythicalLibrarian/$NewShowName/$NewShowName.Ename.txt"&
     echo "$FAired" >> "$mythicalLibrarian/$NewShowName/$NewShowName.FAired.txt"&
     echo "$SeasonNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.S.txt"&
     echo "$EpisodeNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.E.txt"&
     Ename=""
     actualEname=""
     FAired=""
     SeasonNr=""
     EpisodeNr=""

#Get actual show name 
     elif [[ $line == \<EpisodeName\>* ]]; then
      actualEname=`echo "$line" | sed -e s/'<\/EpisodeName>'// -e s/'<EpisodeName>'// -e s/'\&amp\;'/'\&'/ -e s/'\&quot\;'/'\"'/ -e s/'\&amp\;'/'\&'/ -e s/'\&ndash\;'/'-'/ -e s/'\&lt\;'/'\<'/ -e 's/'\&gt\;'/'\>'/' |tr -d '|\?\*\<\"\:\>\+\\\[\]\/'`


#Get OriginalAirDate
    elif [[ $line == \<FirstAired\>* ]]; then
      FAired=`echo "$line" | sed -e s/'<FirstAired>'//g -e s/'<\/FirstAired>'//g`

#Get Season number
     elif [[ $line == \<SeasonNumber\>* ]]; then
      SeasonNr=`echo "$line" |sed -e s/'<SeasonNumber>'// -e s/'<\/SeasonNumber>'//`

#Get Episode number
    elif [[ $line == \<EpisodeNumber\>* ]]; then
      EpisodeNr=`echo "$line" |sed -e 's/<EpisodeNumber>//' -e 's/<\/EpisodeNumber>//'`

    fi
   done < "$mythicalLibrarian/$NewShowName/$NewShowName.xml"


   chmod 777 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".actualEname.txt
   chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".Ename.txt
   chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".FAired.txt
   chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".S.txt
   chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".E.txt
    GotNewInformation=1


  elif [ ! -f "$mythicalLibrarian/$NewShowName/$NewShowName.xml" ]; then
   echo "COULD NOT DOWNLOAD:www.thetvdb.com/api/$APIkey/series/$SeriesID/all/$Language.xml">>"$mythicalLibrarian"/output.log
  fi

Here are some of the data that it processes.

<?xml version="1.0" encoding="UTF-8" ?>
<Data><Series>
  <Actors>|Fred Rogers|Adair Roth|Bert Lloyd|Bud Alder|Carol Saunders|Carole Switala|Deborah Neal Stampo|Don Brockett|Elsie Neal|Emilie Jacobson|Fred Michael|John Reardon|Jos|Judy Rubin|Keith David|Lenny Meledandri|Michael Horton|Robert Trow|Yoshi Ito|</Actors>
  <Airs_DayOfWeek></Airs_DayOfWeek>
  <Airs_Time></Airs_Time>
  <ContentRating></ContentRating>
  <FirstAired>1968-02-01</FirstAired>
  <Genre>|Children|</Genre>
  <Network>PBS</Network>
  <NetworkID></NetworkID>
  <Overview>&quot;In a little toy neighborhood, a tiny trolley rolls past a house at the end of a street.

  <Runtime>30</Runtime>
  <SeriesID>6843</SeriesID>
  <SeriesName>Mister Rogers' Neighborhood</SeriesName>
  <Status>Ended</Status>
  <added></added>
  <addedBy></addedBy>
  <banner>graphical/77750-g.jpg</banner>
  <fanart>fanart/original/77750-1.jpg</fanart>
  <poster></poster>
  <zap2it_id>SH002930</zap2it_id>
</Series>
<Episode>
  <EpisodeName>Change (1)</EpisodeName>
  <EpisodeNumber>1</EpisodeNumber>
  <FirstAired>1968-02-19</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Change (2)</EpisodeName>
  <EpisodeNumber>2</EpisodeNumber>
  <FirstAired>1968-02-20</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Change (3)</EpisodeName>
  <EpisodeNumber>3</EpisodeNumber>
  <FirstAired>1968-02-21</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Change (4)</EpisodeName>
  <EpisodeNumber>4</EpisodeNumber>
  <FirstAired>1968-02-22</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Change (5)</EpisodeName>
  <EpisodeNumber>5</EpisodeNumber>
  <FirstAired>1968-02-23</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 6</EpisodeName>
  <EpisodeNumber>6</EpisodeNumber>
  <FirstAired>1968-02-26</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 7</EpisodeName>
  <EpisodeNumber>7</EpisodeNumber>
  <FirstAired>1968-02-27</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 8</EpisodeName>
  <EpisodeNumber>8</EpisodeNumber>
  <FirstAired>1968-02-28</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 9</EpisodeName>
  <EpisodeNumber>9</EpisodeNumber>
  <FirstAired>1968-02-29</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 10</EpisodeName>
  <EpisodeNumber>10</EpisodeNumber>
  <FirstAired>1968-03-01</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 11</EpisodeName>
  <EpisodeNumber>11</EpisodeNumber>
  <FirstAired>1968-03-04</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 12</EpisodeName>
  <EpisodeNumber>12</EpisodeNumber>
  <FirstAired>1968-03-05</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 13</EpisodeName>
  <EpisodeNumber>13</EpisodeNumber>
  <FirstAired>1968-03-06</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 14</EpisodeName>
  <EpisodeNumber>14</EpisodeNumber>
  <FirstAired>1968-03-07</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 15</EpisodeName>
  <EpisodeNumber>15</EpisodeNumber>
  <FirstAired>1968-03-08</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Welcome Donkey Hodie (1)</EpisodeName>
  <EpisodeNumber>16</EpisodeNumber>
  <FirstAired>1968-03-11</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Welcome Donkey Hodie (2)</EpisodeName>
  <EpisodeNumber>17</EpisodeNumber>
  <FirstAired>1968-03-12</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Welcome Donkey Hodie (3)</EpisodeName>
  <EpisodeNumber>18</EpisodeNumber>
  <FirstAired>1968-03-13</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Welcome Donkey Hodie (4)</EpisodeName>
  <EpisodeNumber>19</EpisodeNumber>
  <FirstAired>1968-03-14</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Welcome Donkey Hodie (5)</EpisodeName>
  <EpisodeNumber>20</EpisodeNumber>
  <FirstAired>1968-03-15</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 21</EpisodeName>
  <EpisodeNumber>21</EpisodeNumber>
  <FirstAired>1968-03-18</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 22</EpisodeName>
  <EpisodeNumber>22</EpisodeNumber>
  <FirstAired>1968-03-19</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 23</EpisodeName>
  <EpisodeNumber>23</EpisodeNumber>
  <FirstAired>1968-03-20</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 24</EpisodeName>
  <EpisodeNumber>24</EpisodeNumber>
  <FirstAired>1968-03-21</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 25</EpisodeName>
  <EpisodeNumber>25</EpisodeNumber>
  <FirstAired>1968-03-22</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 26</EpisodeName>
  <EpisodeNumber>26</EpisodeNumber>
  <FirstAired>1968-03-25</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 27</EpisodeName>
  <EpisodeNumber>27</EpisodeNumber>
  <FirstAired>1968-03-26</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 28</EpisodeName>
  <EpisodeNumber>28</EpisodeNumber>
  <FirstAired>1968-03-27</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 29</EpisodeName>
  <EpisodeNumber>29</EpisodeNumber>
  <FirstAired>1968-03-28</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 30</EpisodeName>
  <EpisodeNumber>30</EpisodeNumber>
  <FirstAired>1968-03-29</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Red Monster (1)</EpisodeName>
  <EpisodeNumber>31</EpisodeNumber>
  <FirstAired>1968-04-01</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Red Monster (2)</EpisodeName>
  <EpisodeNumber>32</EpisodeNumber>
  <FirstAired>1968-04-02</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Red Monster (3)</EpisodeName>
  <EpisodeNumber>33</EpisodeNumber>
  <FirstAired>1968-04-03</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Red Monster (4)</EpisodeName>
  <EpisodeNumber>34</EpisodeNumber>
  <FirstAired>1968-04-04</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Red Monster (5)</EpisodeName>
  <EpisodeNumber>35</EpisodeNumber>
  <FirstAired>1968-04-05</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 36</EpisodeName>
  <EpisodeNumber>36</EpisodeNumber>
  <FirstAired>1968-04-08</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 37</EpisodeName>
  <EpisodeNumber>37</EpisodeNumber>
  <FirstAired>1968-04-09</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 38</EpisodeName>
  <EpisodeNumber>38</EpisodeNumber>
  <FirstAired>1968-04-10</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 39</EpisodeName>
  <EpisodeNumber>39</EpisodeNumber>
  <FirstAired>1968-04-11</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 40</EpisodeName>
  <EpisodeNumber>40</EpisodeNumber>
  <FirstAired>1968-04-12</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 41</EpisodeName>
  <EpisodeNumber>41</EpisodeNumber>
  <FirstAired>1968-04-15</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 42</EpisodeName>
  <EpisodeNumber>42</EpisodeNumber>
  <FirstAired>1968-04-16</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 43</EpisodeName>
  <EpisodeNumber>43</EpisodeNumber>
  <FirstAired>1968-04-17</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 44</EpisodeName>
  <EpisodeNumber>44</EpisodeNumber>
  <FirstAired>1968-04-18</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 45</EpisodeName>
  <EpisodeNumber>45</EpisodeNumber>
  <FirstAired>1968-04-19</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 46</EpisodeName>
  <EpisodeNumber>46</EpisodeNumber>
  <FirstAired>1968-04-22</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 47</EpisodeName>
  <EpisodeNumber>47</EpisodeNumber>
  <FirstAired>1968-04-23</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 48</EpisodeName>
  <EpisodeNumber>48</EpisodeNumber>
  <FirstAired>1968-04-24</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 49</EpisodeName>
  <EpisodeNumber>49</EpisodeNumber>
  <FirstAired>1968-04-25</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 50</EpisodeName>
  <EpisodeNumber>50</EpisodeNumber>
  <FirstAired>1968-04-26</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 51</EpisodeName>
  <EpisodeNumber>51</EpisodeNumber>
  <FirstAired>1968-04-29</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 52</EpisodeName>
  <EpisodeNumber>52</EpisodeNumber>
  <FirstAired>1968-04-30</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 53</EpisodeName>
  <EpisodeNumber>53</EpisodeNumber>
  <FirstAired>1968-05-01</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 54</EpisodeName>
  <EpisodeNumber>54</EpisodeNumber>
  <FirstAired>1968-05-02</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 55</EpisodeName>
  <EpisodeNumber>55</EpisodeNumber>
  <FirstAired>1968-05-03</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 56</EpisodeName>
  <EpisodeNumber>56</EpisodeNumber>
  <FirstAired>1968-05-06</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 57</EpisodeName>
  <EpisodeNumber>57</EpisodeNumber>
  <FirstAired>1968-05-07</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 58</EpisodeName>
  <EpisodeNumber>58</EpisodeNumber>
  <FirstAired>1968-05-08</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 59</EpisodeName>
  <EpisodeNumber>59</EpisodeNumber>
  <FirstAired>1968-05-09</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 60</EpisodeName>
  <EpisodeNumber>60</EpisodeNumber>
  <FirstAired>1968-05-10</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 61</EpisodeName>
  <EpisodeNumber>61</EpisodeNumber>
  <FirstAired>1968-05-13</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 62</EpisodeName>
  <EpisodeNumber>62</EpisodeNumber>
  <FirstAired>1968-05-14</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 63</EpisodeName>
  <EpisodeNumber>63</EpisodeNumber>
  <FirstAired>1968-05-15</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 64</EpisodeName>
  <EpisodeNumber>64</EpisodeNumber>
  <FirstAired>1968-05-16</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 65</EpisodeName>
  <EpisodeNumber>65</EpisodeNumber>
  <FirstAired>1968-05-17</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 66</EpisodeName>
  <EpisodeNumber>66</EpisodeNumber>
  <FirstAired>1968-05-20</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 67</EpisodeName>
  <EpisodeNumber>67</EpisodeNumber>
  <FirstAired>1968-05-21</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 68</EpisodeName>
  <EpisodeNumber>68</EpisodeNumber>
  <FirstAired>1968-05-22</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 69</EpisodeName>
  <EpisodeNumber>69</EpisodeNumber>
  <FirstAired>1968-05-23</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 70</EpisodeName>
  <EpisodeNumber>70</EpisodeNumber>
  <FirstAired>1968-05-24</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 71</EpisodeName>
  <EpisodeNumber>71</EpisodeNumber>
  <FirstAired>1968-05-27</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
  <EpisodeName>Show 72</EpisodeName>
  <EpisodeNumber>72</EpisodeNumber>
  <FirstAired>1968-05-28</FirstAired>
  <SeasonNumber>1</SeasonNumber>
</Episode>

The problem is that on the i7 processor it takes 14.5 seconds. This is about 10 times slower in my media center. I tried using a case statement that takes 15 seconds on a fast processor.

I would like to know how to speed up this process. It seems like it's ridiculously slow for bash, which is supposed to be intended for data manipulation and file operations.

+3
source share
6 answers

You will get significant acceleration, dropping &from the end of all these operators echo.

Test1:

$ time { for i in {1..1000}; do echo "hello"& done >/dev/null; } | cat

real    0m10.357s
user    0m2.764s
sys     0m15.441s

cat "done", . cat "" -. , , , .

Test2:

$ time { for i in {1..1000}; do echo "hello"; done >/dev/null; }

real    0m0.152s
user    0m0.132s
sys     0m0.020s

, .

Bash regex .

:

elif [[ $line == \<EpisodeName\>* ]]; then
    actualEname=${line//<\/EpisodeName>/}
    actualEname=${actualEname//<EpisodeName>/}
    actualEname=${actualEname//&amp;/&}
    actualEname=${actualEname//&ndash;/-}
    for string in '|' '&lt;' '&gt;' '&quot;' '?' '*' '<' '>' ':' '"' '+' '\' '[' ']' '/'
    do
        actualEname=${actualEname//$string}
    done

&amp; , . , HTML, . ? g (global).

Test3:

$ time { for i in {1..100}; do
    line='<EpisodeName>&lt;foo&amp;bar&ndash;baz&gt;Season&ndash;3&ndash;&quot;quux&quot;?*<>:"+\[]/</EpisodeName>'
    actualEname=$(echo "$line" | sed -e 's/<\/EpisodeName>//' -e 's/<EpisodeName>//' -e 's/&amp;/\&/g' -e 's/&quot;/"/g' -e 's/&ndash;/-/g' -e 's/&lt;/</g' -e 's/&gt;/>/g' |tr -d '|?*<":>+\\[]/')
done; }

real    0m7.779s
user    0m3.164s
sys     0m5.436s

Test4:

$ time { for i in {1..100}; do
    line='<EpisodeName>&lt;foo&amp;bar&ndash;baz&gt;Season&ndash;3&ndash;&quot;quux&quot;?*<>:"+\[]/</EpisodeName>
    actualEname=${line//<\/EpisodeName>/}
    actualEname=${actualEname//<EpisodeName>/}
    actualEname=${actualEname//&amp;/&}
    actualEname=${actualEname//&ndash;/-}
    for string in '|' '&lt;' '&gt;' '&quot;' '\?' '\*' '<' '>' ':' '"' '+' '\\' '[' ']' '\/'
    do
        actualEname=${actualEname//$string}
    done
done; }

real    0m5.403s
user    0m2.492s
sys     0m2.960s
+4

- XMLStarlet, XML.

+1

, , , script (sed, tr).

, XML . bash, -, XSLT, XML , , .

Perl.

0

BASH, , , .

BASH . - * sh, .

Python or Perl would be a much better choice for problem space.

0
source

I just tried this:

        echo "Parsing Downloaded information: $NewShowName.xml "
        while read line
        do




            if [[ $line == \<\/Episode\> ]]; then
                (( ++recordNumber ))
                echo -ne "Building Record:$recordNumber ${actualEname:0:20}            \r" 1>&2 
                echo "$EpisodeName" >> "$mythicalLibrarian/$NewShowName/$NewShowName.actualEname.txt"&
                Ename=`echo "$actualEname" |sed 's/;.*//'`
                echo "$EpisodeName" >> "$mythicalLibrarian/$NewShowName/$NewShowName.Ename.txt"&
                echo "$FirstAired" >> "$mythicalLibrarian/$NewShowName/$NewShowName.FAired.txt"&
                echo "$SeasonNumber" >> "$mythicalLibrarian/$NewShowName/$NewShowName.S.txt"&
                echo "$EpisodeNumber" >> "$mythicalLibrarian/$NewShowName/$NewShowName.E.txt"&
                EpisodeName=""
                actualEname=""
                FirstAired=""
                SeasonNumber=""
                EpisodeNumber=""
            else 
                var=`echo $line |tr '<>' ' '|awk '{print $1}'`

                value=`echo "$line"|sed -e s/'<'"$var"'>'// -e s/'<\/'"$var"'>'// -e s/'\&amp\;'/'\&'/ -e s/'\&quot\;'/'\"'/ -e s/'\&amp\;'/'\&'/ -e s/'\&ndash\;'/'-'/ -e s/'\&lt\;'/'\<'/ -e 's/'\&gt\;'/'\>'/' |tr -d '|\?\*\<\"\:\>\+\\\[\]\/'`
                eval $var="'$value'"
            fi

It took 43 seconds on a faster processor

0
source

Holy cow Dennis Williamson, she analyzes in less than 1/2 second. It just flickers across the screen. It used to take 15 seconds, but now it is so fast that I can’t even say that this is happening.

These are the changes that Dennis Williamson proposed. I just post it here.

            echo "Parsing Downloaded information: $NewShowName.xml "
            while read line
            do

                if [[ $line == \<\/Episode\> ]]; then
                    (( ++recordNumber ))
                    echo -ne "Building Record:$recordNumber ${actualEname:0:20}           \r" 1>&2 
                    echo "$actualEname" >> "$mythicalLibrarian/$NewShowName/$NewShowName.actualEname.txt"

                    echo "$Ename" >> "$mythicalLibrarian/$NewShowName/$NewShowName.Ename.txt"
                    echo "$FAired" >> "$mythicalLibrarian/$NewShowName/$NewShowName.FAired.txt"
                    echo "$SeasonNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.S.txt"
                    echo "$EpisodeNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.E.txt"
                    Ename=""
                    actualEname=""
                    FAired=""
                    SeasonNr=""
                    EpisodeNr=""

#Get actual show name   
                elif [[ $line == \<EpisodeName\>* ]]; then
                    line=${line/<\/EpisodeName>/}
                    line=${line/<EpisodeName>/}
                    line=${line/&lt;}
                    line=${line/&gt;/} 
                    line=${line/&quot;/} 
                    line=${line/&amp;/&}
                    line=${line/\|/}
                    line=${line/\?/}
                    line=${line/\*/}
                    line=${line/\:/}
                    line=${line/\+/}
                    line=${line/\\/}
                    line=${line/\//}
                    line=${line/\[/}
                    line=${line/\]/}
                    line=${line/\'/}
                    line=${line/\"/}
                    actualEname=${line/&ndash;/-}
                    Ename=${actualEname/;*/}

#Get OriginalAirDate
                elif [[ $line == \<FirstAired\>* ]]; then
                    line=${line/<\/FirstAired>/}
                    line=${line/<FirstAired>/}
                    FAired=$line

#Get Season number
                elif [[ $line == \<SeasonNumber\>* ]]; then
                    line=${line/<\/SeasonNumber>/}
                    line=${line/<SeasonNumber>/}
                    SeasonNr=$line

#Get Episode number
                elif [[ $line == \<EpisodeNumber\>* ]]; then
                    line=${line/<\/EpisodeNumber>/}
                    line=${line/<EpisodeNumber>/}
                    EpisodeNr=$line
                fi
            done < "$mythicalLibrarian/$NewShowName/$NewShowName.xml"


            chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".actualEname.txt
            chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".Ename.txt
            chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".FAired.txt
            chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".S.txt
            chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".E.txt
            GotNewInformation=1
0
source

Source: https://habr.com/ru/post/1776996/

More articles:


All Articles