Why shouldn't you use YAML to store eye tracking data? /s
-
This post did not contain any content.
Also let's represent all numbers in scientific notation, I'm sure that's going to make it easier to read...
-
This post did not contain any content.
This is nasty to look at
-
This post did not contain any content.wrote on last edited by [email protected]
Why you shouldn't use YAML
-
Why you shouldn't use YAML
The best approach would be to never use yaml for anything
-
I think SQLite is a great middle ground. It saves the database as a single .db file, and can do everything an SQL database can do. Querying for data is a lot more flexible and a lot faster. The tools for manipulating the data in any way you want are very good and very robust.
However, I'm not sure how it would affect file size. It might be smaller because JSON/YAML wastes a lot of characters on redundant information (field names) and storing numbers as text, which the database would store as binary data in a defined structure. On the other hand, extra space is used to make common SQL operations happen much faster using fancy data structures. I don't know which effect is greater so file size could be bigger or smaller.
I didn't look to much at the data but I think csv might actually be an appropriate format for this?
Nice simple plaintext and very easy to parse into a datastructure for analysing/using it in python or similar
-
This post did not contain any content.
I’d probably just use line delimited JSON or CSV for this use case. It plays nicely with cat and other standard tools and basically all the yaml is doing is wrapping raw json and adding extra parse time/complexity.
In the end consider converting this to parquet for analysis, you probably won’t get much from compression or row-group clustering, but you will get benefits from the column store format when reading the data.
-
This post did not contain any content.
I'm amazed at developers who don't grasp that you don't need to have absolutely everything under the sun in a human readable file format. This is such a textbook case...
-
I’d probably just use line delimited JSON or CSV for this use case. It plays nicely with cat and other standard tools and basically all the yaml is doing is wrapping raw json and adding extra parse time/complexity.
In the end consider converting this to parquet for analysis, you probably won’t get much from compression or row-group clustering, but you will get benefits from the column store format when reading the data.
wrote on last edited by [email protected]Thanks for the advice, but this is just the format of some eyetracking software I had to use not something I develop myself
-
This post did not contain any content.
If you’re using a library to handle deserialization , the ugliness of the serial format doesn’t matter that much.
Just call yaml.load() and forget about it.
-
This isn't YAML, this is just sparkling JSON
All yaml is just sparkling JSON.
-
Thanks for the advice, but this is just the format of some eyetracking software I had to use not something I develop myself
Ah, well, such is software dependencies.
-
All yaml is just sparkling JSON.
Always has been
-
I'm amazed at developers who don't grasp that you don't need to have absolutely everything under the sun in a human readable file format. This is such a textbook case...
Even if you want it to be human readable, you don't need to include the name into every field and use balanced separators.
Any CSV variant would be an improvement already.
-
I'm amazed at developers who don't grasp that you don't need to have absolutely everything under the sun in a human readable file format. This is such a textbook case...
That's it everyone, back to copybooks.
-
I think SQLite is a great middle ground. It saves the database as a single .db file, and can do everything an SQL database can do. Querying for data is a lot more flexible and a lot faster. The tools for manipulating the data in any way you want are very good and very robust.
However, I'm not sure how it would affect file size. It might be smaller because JSON/YAML wastes a lot of characters on redundant information (field names) and storing numbers as text, which the database would store as binary data in a defined structure. On the other hand, extra space is used to make common SQL operations happen much faster using fancy data structures. I don't know which effect is greater so file size could be bigger or smaller.
SQLite would definitely be smaller, faster, and require less memory.
Thing is, it's 2025, roughly 20 years since anybody's given half a shit about storage efficiency, memory efficiency, or even CPU efficiency for anything so small. Presumably this is not something they need to query dynamically.
-
Why you shouldn't use YAML
YAML doesn't require any level of accuracy for floating point numbers, and that doc appears to have numbers large enough to run into problems for single-precision floats (maybe double too). That means different parsers could give you different results.
-
If you’re using a library to handle deserialization , the ugliness of the serial format doesn’t matter that much.
Just call yaml.load() and forget about it.
That works until you realize your calculations are all wrong due to floating point inaccuracies. YAML doesn't require any level of precision for floats, so different parsers on a document may give you different results.
-
That works until you realize your calculations are all wrong due to floating point inaccuracies. YAML doesn't require any level of precision for floats, so different parsers on a document may give you different results.
What text based serialization formats do enforce numeric precision?
AFAIK it’s always left up to the writer (serializer)
-
SQLite would definitely be smaller, faster, and require less memory.
Thing is, it's 2025, roughly 20 years since anybody's given half a shit about storage efficiency, memory efficiency, or even CPU efficiency for anything so small. Presumably this is not something they need to query dynamically.
True (in most contexts, probably including this one), but I think that only makes the case for SQLite stronger. What people do still care about is a good flexible, usable and reliable interface. I'm not sure how to get that with YAML.
-
What text based serialization formats do enforce numeric precision?
AFAIK it’s always left up to the writer (serializer)
wrote on last edited by [email protected]Cuelang: https://cuelang.org/docs/reference/spec/#numeric-values
Implementation restriction: although numeric values have arbitrary precision in the language, implementations may implement them using an internal representation with limited precision. That said, every implementation must:
- Represent integer values with at least 256 bits.
- Represent floating-point values with a mantissa of at least 256 bits and a signed binary exponent of at least 16 bits.
- Give an error if unable to represent an integer value precisely.
- Give an error if unable to represent a floating-point value due to overflow.
- Round to the nearest representable value if unable to represent a floating-point value due to limits on precision. These requirements apply to the result of any expression except for builtin functions, for which an unusual loss of precision must be explicitly documented.
