RoboTask forum

How Do I Go About Removing Duplicates in a TXT File?

Example:

{Version 1}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">

[facebook http://video.ak.facebook.com/cfs-ak-
snc6/78679/449/109340239134123_34453.mp4]

[facebook http://video.ak.facebook.com/cfs-ak-
snc6/78675/341/109340309134116_56217.mp4]


{Version 2}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78661/667/109152589155101_11180.mp4]

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78605/484/109152622488431_3792.mp4]


{Version 3}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78661/667/109152589155101_11180.mp4]

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78605/484/109152622488431_3792.mp4]


{Version 4}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78661/667/109152589155101_11180.mp4]

[facebook http://video.ak.fbcdn.net/cfs-ak-
snc4/78605/484/109152622488431_3792.mp4]



{Version 5}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">
[megavideo LH3KYZAE]



{Version 6}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">
[zshare http://www.zshare.net/videoplayer/player.php?
SID=dl023&FID=83015550&FN=Dragon Ball Kai
82.flv]

{Version 7}
<img src="http://www.myanimelinks.com/wp-
content/uploads/2010/03/engsub.png">
[novamov 0ye167blqu6b1]


Version 2, 3, & 4 are the same, how can i remove them and
at the same time rename the other versions accordingly?
From Version 1-4 in order.

I've just edited the post with more information on what I
need to do. Thanks Oleg, I'm awaiting your response.

I've been playing with this example here:

http://robotask.com/forum/viewtopic.php ... 975&p=3127

However I can't seem to figure it out, It removes
everything that is the same from the first row of
information. The output is not quite what i'm looking
for. I wanted this format I posted above to stay the
same, just the versions that are duplicate to be removed
and the versions to be renamed accordingly, from 1-4, for
this example. Each example is different, as each episode
is different. I've managed to make it very far with your
program, I just need your help here to customize this
last part of code.

I've been playing with this example here:

http://www.robotask.com/forum/forum_posts.asp?
TID=975&KW=double

Yes, algorithm should be the similar. But you need to compare the whole strings

How Do I Go About Doing This Oleg? I've managed to remove
all duplicates however it always leaves the versions with
parts with only one part. Also It removes all the
{tab=Version 1} codes, and the Sub image code. I'm stuck
here...need your help.

Did you try to use some database?
For example - MSAccess.

If you want to compare some text blocks which contain several strings, simple text file is very uncomfortable format.

Also you can write your information (image address and subcodes) into the comma-separated text.
See "Delimit variable" action. It helps you to make correct comma-separated string.
1-st column - address
2-nd - subcode 1
3-d - subcode 2
etc

Thus, you write your information as a table and you can apply the previous algorithm (see here)

Hello Oleg,

I did not use a database, it's in a simple txt file. I
understand the idea of using the csv however I don't know
how to implement it. The program I came up with just
outputs the codes in a simple txt file. I am trying to
think of a way to go about removing the duplicates and
retain the information I want, however it seems too complex
of a code. Can you try to make a code that works for me?
Please check my first post on this thread for the codes.
Thank you so very much! I really appreciate your efforts!