[selyb]

I want to set up something like hard links but only to parts of a file. I have searched all over for something like this or maybe a versioning file system that doesn't make complete copies of the original file.

Here is my scenario: I have a folder with music albums. I want to add metadata to these but I want to keep the original unmodified files as well. I'm searching for some way to be able to access either file versions but I really don't like having to keep all the redundant data around.

[bphlpt]

The only way I know to do this is to keep two versions of the file, with different file names, one with the metadata and one without, which I think will totally defeat what you are trying to accomplish. Sorry.

Cheers and Regards

[selyb]

That is what I'm doing now. I have duplicates of a couple gigs of flac albums. I could also think of a few other uses for such a system

[bphlpt]

What I don't get though, is if you have a set of the files with the metadata info embedded, what purpose does a set of the files without the metadata really serve? In other words, why do you want to keep a copy of the original unmodified files? You have explained what you want to do, but not why. Besides, if you ever need to remove the metadata from the files, though I'm not sure why you would, you could use JPEG.&.PNG.Stripper - http://www.steelbytes.com/?mid=30 - a free tool that can strip the metadata from the files while preserving the file date and times.

Quote

A tool for stripping/cleaning/removing uncessary metadata (junk) from JPG/JPEG/JFIF & PNG files. The image quality IS NOT AFFECTED. Includes command line support. Just specify a folder or file on the commandline (wildcards allowed)

EDIT: LOL I'm going to have to get my glasses fixed.

Cheers and Regards

This post has been edited by bphlpt: 20 February 2013 - 02:37 AM

[jaclaz]

@bphlpt
Are music albums recorded to JPEG's or PNG's?


@selyb
More seriously, which format are those "music albums" stored in? .flac. right?
http://flac.sourcefo...net/format.html
then:
http://xiph.org/flac...s_metaflac.html

The issue is really about possible bugs in metaflac (or similar tools), in theory it is possible to strip (saving them to files) the metadata and later re-build the original re-adding the metadata.

Remember :

Quote

In theory there is no difference between theory and practice, but in practice there is.

jaclaz

This post has been edited by jaclaz: 19 February 2013 - 01:43 PM

[iamtheky]

Or maintain the data in an ini. The following autoitscript will facilitate building an ini of data for whichever file you choose. This seems like it would be easier than maintaining copies of entire files, especially if you are not importing the metadata and entering it yourself anyway.

Just select a file, then add a Key (e.g. "Artist") , then add the data (e.g. "firstname lastname"). On subsequent runs if data already exists for the file selected it will display that data prior to the inputboxes. It will build the ini in the directory the script is ran from.

#include<array.au3>

$file = FileOpenDialog("Select File" , @ScriptDir , "ALL(*.*)" , 1)
$aINI = inireadsection(@ScriptDir & "\metadata.ini" , $file)
	If @Error <> 1 Then _Arraydisplay ($aINI)
$key = inputbox("Add Key" , "What type of data is this (one word)" , "Artist")
	If @Error = 1 Then Exit
$input = inputbox("Add Data" , "Add the Data" , "FirstName LastName")
iniwrite(@ScriptDir & "\metadata.ini" , $file , $key , $input)

This post has been edited by iamtheky: 19 February 2013 - 02:35 PM

[selyb]

I need the unmodified tracks for bittorrent but they don't have ReplayGain info in the metadata so I keep a separate copy of the entire album with RG tags. Also, some have the album cover injected into each track. I remove this and use a single folder.jpg.

This post has been edited by selyb: 19 February 2013 - 05:51 PM

[jumper]

Quicktime allows you to create a small, new media file with metadata and just a reference to the music track.

[allen2]

So if i understood properly, you need to have one copy of your album with the folder.jpg and not the other or are there other files you don't need on the copy ?
If it's on the same hard drive and different files, hard links would work but only in this case. Other than that, you'll need most likely a tailor made tool to do what you need.

[selyb]

I'm not sure if I could explain differently :-/ I'll try a different situation

Let's say I have a small business and I install Linux distros on client's computers. For the good of the community, I share these distros back using bittorrent. Let's say that in every one of the ISO's I like to insert my company logo as a default desktop background and the most efficient way to do this is to modify the ISO. In this scenario, I may have 50 different 4.6GB ISOs that I share on bittorrent and 50 more that are my customized versions. The difference between each original and it's customized version is only a couple megabytes so 99% of the data is the same between each pair. Bittorrent is not going to ignore any little part so there is no way (that I know of) to share only that 99%.

The solution I had in mind is by using something like a small diff patch, I could give programs access to two different files while most of it points to the same data. Currently, I use hard links to the same data from multiple file names. What I need is some way to (for example) have "c:\somefolder\somefilename" point to "c:\otherfolder\otherfile" with "c:\patchdata\otherfilepatch". In this way, Bittorrent could share the original file and Imgburn could access the modified file but I don't have to lose nearly 50x4.6GB of redundant hdd space.

[bphlpt]

jaclaz, on 19 February 2013 - 01:42 PM, said:

@bphlpt
Are music albums recorded to JPEG's or PNG's?

Touche! I'm an id10t.

Cheers and Regards

[allen2]

To use the iso example, then i would create a script/program that would be able to monitor iso change (modified date) then extract all the files except the uneeded ones (to follow your example the desktop back ground) and then recreate the iso. But this example show best that being able to create another iso file from the first doesn't avoid the double space usage. The only proper solution in this case would be to extract all files from iso, use hard link to duplicate them and store them in this form so out of the iso container and use a script/program that would handle files to iso transparently (without having to create the iso file).
Edit: fixed mistakes and missing words.

This post has been edited by allen2: 20 February 2013 - 03:30 AM

[jaclaz]

Yes/no.
The "patch" route is not really feasible/useful/smart.

The "patch" is "smart" when you have a binary file and you have to replace a given set of bytes with another one (SAME number of bytes).

Size of the file before and after patch is the same.

In the case of .iso you can "simulate" the patch by replacing a whole file with an empty file (or a modified one) of the same length.
(as an example, Winimage can do this)
This allows to NOT alter in any way the filesystem indexing tables.
Otherwise you need a .iso editor (let's say MagicIso or PowerIso or the like), i.e. a tool that rebuilds the indexing. (and that from experience works - say - 95% of the times BUT it is not totally error free).

The case of any file which includes metadata is very similar to the .iso one, you can do the same by replacing the "filled" metadata fields with empty ones, BUT you cannot remove or add a metadata field without re-indexing/re-creating the whole file (otherwise a tool like metaflac would have NO reason to exist).

bphlpt, on 20 February 2013 - 02:37 AM, said:

jaclaz, on 19 February 2013 - 01:42 PM, said:

@bphlpt
Are music albums recorded to JPEG's or PNG's?

Touche! I'm an id10t.

Cheers and Regards

Naaaah, let's fake that you were thinking of music albums covers only

jaclaz

This post has been edited by jaclaz: 20 February 2013 - 08:47 AM

[submix8c]

http://en.wikipedia.org/wiki/Xdelta
(link to homepage at bottom)

Works fine - any size difference. Generate DeltaFile, delete Original, run DeltaFile against Modified to reproduce Original. Bear in mind the HDD FileDates will not be retained so you'll have to keep a log of that and a Date changer program.

Syntax -
http://code.google.c...mmandLineSyntax

HTH

This post has been edited by submix8c: 20 February 2013 - 11:16 AM

[jaclaz]

submix8c, on 20 February 2013 - 11:09 AM, said:

Works fine - any size difference. Generate DeltaFile, delete Original, run DeltaFile against Modified to reproduce Original. Bear in mind the HDD FileDates will not be retained so you'll have to keep a log of that and a Date changer program.

Of course there is no issue with making an Xdelta between two files even if they have different sizes .

But you don't have available at the same time the original and the patched file. (unless you keep the original ).

I mean OP now has 2 Gb of files (originals) AND (almost) 2 Gb of files (stripped) = total (almost) 4 Gb.
By making delta files for all the originals he will end up with (almost) 2 Gb (stripped) and (say) 50 Mb of xdelta files = total (roughly) 2 Gb.
He needs to have the (almost) 2 Gb of stripped files "available/accessible", but everytime he needs one of the "originals" he will have to recreate it on-the-fly applying to the stripped file the xdelta patch.

What I got from the original request was *something* that allowed to have AT THE SAME TIME available the "original" and the "stripped" versions of all files BUT saving the space.
This cannot be done with "ordinary" means.
What the OP asked for (again IF I got it right) is *something* similar to these:
http://khenriks.github.com/mp3fs/
http://blog.einval.c...ware/fuse-music
which I don't think actually exists or exists for Windows (though possibly it can be ported through Dokan ) or will do specifically .

jaclaz

[bphlpt]

jaclaz, on 20 February 2013 - 11:46 AM, said:

I mean OP now has 2 Gb of files (originals) AND (almost) 2 Gb of files (stripped) = total (almost) 4 Gb.
...
What I got from the original request was *something* that allowed to have AT THE SAME TIME available the "original" and the "stripped" versions of all files BUT saving the space.

I agree with your understanding that he want's both versions available at the same time, but I think you have it slightly backwards. I the think the originals ARE the stripped versions. The ones he adds have metadata ADDED to them.

selyb, on 19 February 2013 - 12:24 AM, said:

I want to add metadata to these but I want to keep the original unmodified files as well.

Not that that makes any difference to your conclusions. I don't think that is currently possible to do, though I can see it would be handy. As of now, an external file with the added info, or "duplicate" copies as he is doing now, seems to be his only options.

Cheers and Regards

[selyb]

Jaclaz is correct on all accounts. Another solution I thought of is to have some way to point to two files as if they were one file

• file1 -> partA + partC
  
• file2 -> partB + partC

BTW, working on windows would be a perk but is not required. I can install the solution on a linux box or VM and share the folder(s) with RW permissions.

[jumper]

The original FLAC files without metadata already exist, so it's just a matter of generating files or streams that also contain the metadata without duplicating the original data.

IIRC, VideoLan and VirtualDub both have a video/audio server mode that can create streams on-the-fly from separate audio and metadata files. And Quicktime / MP4 file formats support metadata physically present in the file along with audio tracks that are referenced from another file (the original FLAC file in our case here). Perhaps there are also other ASX or playlist formats that would suffice.

[jaclaz]

The posted FUSE filesystem drivers approach actually does provide the solution, you have an "original" series of .flac files that are actually available ALSO AND AT THE SAME TIME as .mp3.
The shortcoming of this approach is that decoding/re-encoding the file on-the-fly (and upon request) will need some (a lot of) computer power.

Think (very simplified) to gzip.
The gzip compression format is "pipable", so you can well keep your files gzipped, and have them uncompressed on the fly transparently, but this saves space at the cost of CPU cycles.

It is possible in theory to make a .flac FUSE filesystem capable of stripping on the fly the metadata, which should be fairly simple and not hogging the CPU, but I don't think there is one already made:
http://sourceforge.n...tle=FileSystems

Another couple examples:
http://amaras-tech.c..._flac_to_mov_fs
https://www.uitwisse...projects/yacufs <- this one is seemingly dead
http://cuemounter.codeplex.com/ <- Windows

About the:

• file1 -> partA + partC
  
• file2 -> partB + partC

IF the metadata is contained exclusively in the first part (or - if you prefer - part C is NOT metadata) it is also possible, though again I don't think something already exists.

Just for the record, here is that concept applied to the whole filesystem (nothing you can actually use, just a "demo" or PoC), meet Overlapping FileSystem :
http://reboot.pro/to...ub4dos/?p=19803
(idea was never actually finalized as there was little interest and it 's use would have been however useful only for very "narrow" scopes)

jaclaz

[selyb]

I think maybe I can install svn + webdav on my server with apache and map a drive letter to it.