Discussion:
Rainbow tables conversaton
(too old to reply)
Вадим Яницкий
2016-01-17 09:23:00 UTC
Permalink
Raw Message
Hi folks!

I'm new in A5/1 world and I have read that 2 Tb is not enough for table
conversation.

But I have a server with 2 Tb and 60 Gb of additional space only.
My question is: is it possible to convert the tables one by one? I.e. can I
download first table (about 41 Gb) on my 60 Gb HDD and convert it, download
next one and convert with offset ...?

And one more question. Can anyone explain me the Behemoth.py configuration
format? What means this section: "#Tables: dev id(advance) offset"? Maybe
it is an answer to my previous question. :)

With best regards,
Vadim Yanitskiy.
Jan Hrach
2016-01-17 11:49:26 UTC
Permalink
Raw Message
I'm new in A5/1 world and I have read that 2 Tb is not enough for table conversation.
It is (you can convert it one-by-one, requiring only 40 additional GB). Though I'd better keep the original files in case you find out something is screwed up.
See https://brmlab.cz/project/gsm/deka/deka-admin
Can anyone explain me the Behemoth.py configuration format? What means this section: "#Tables: dev id(advance) offset"?
The hash-mark lines are the comments.
I think you want to tell it somehow how many tables should be installed on which device, but that's not the case with one-by-one installation.

If you have it on filesystem as files, I'd just run
./TableConvert di $file.dlt $file.ins:0 $file.idx
on every file. You can do the block device installation later with dd seek=X.
Hi folks!
I'm new in A5/1 world and I have read that 2 Tb is not enough for table conversation.
But I have a server with 2 Tb and 60 Gb of additional space only.
My question is: is it possible to convert the tables one by one? I.e. can I download first table (about 41 Gb) on my 60 Gb HDD and convert it, download next one and convert with offset ...?
And one more question. Can anyone explain me the Behemoth.py configuration format? What means this section: "#Tables: dev id(advance) offset"? Maybe it is an answer to my previous question. :)
With best regards,
Vadim Yanitskiy.
_______________________________________________
A51 mailing list
https://lists.srlabs.de/cgi-bin/mailman/listinfo/a51
--
Jan Hrach | http://jenda.hrach.eu/
GPG CD98 5440 4372 0C6D 164D A24D F019 2F8E 6527 282E
Вадим Яницкий
2016-01-18 16:12:03 UTC
Permalink
Raw Message
Thanks for explanations!

Though I'd better keep the original files in case you find out something is
Post by Jan Hrach
screwed up.
It may sound sad, but I have no more than 2 Tb of space.
I have no even direct HDD access because my server is virtualized and
running on the Hyper-V (it's not my own).

If I understood correctly, I need to do the following actions:
1) Prepare 2 Tb HDD changing the ownership using `chown`.
1.1) In case of file system storage-method format it to ext4, for example.
2) Download the first part (a51_table_100.dlt) to the home folder
of my "buffer-hdd" (60 Gb).
3.1) In case of file system storage-method I should just run:
./TableConvert di (/dev/sda)/home/$file.dlt \
(/dev/sdb)/$file.ins:0 (/dev/sda)/home/kraken/indexes/$file.idx
on every downloaded file. And in this case the offset param will always = 0.
3.2) In case of RAW storage-method there is no file system and I have to
calculate offset for every new table. How can I do that?

Is this sequence of actions right?

With best regards,
Vadim Yanitsky.
Post by Jan Hrach
Post by Вадим Яницкий
I'm new in A5/1 world and I have read that 2 Tb is not enough for table
conversation.
It is (you can convert it one-by-one, requiring only 40 additional GB).
Though I'd better keep the original files in case you find out something is
screwed up.
See https://brmlab.cz/project/gsm/deka/deka-admin
Post by Вадим Яницкий
Can anyone explain me the Behemoth.py configuration format? What means
this section: "#Tables: dev id(advance) offset"?
The hash-mark lines are the comments.
I think you want to tell it somehow how many tables should be installed on
which device, but that's not the case with one-by-one installation.
If you have it on filesystem as files, I'd just run
./TableConvert di $file.dlt $file.ins:0 $file.idx
on every file. You can do the block device installation later with dd
seek=X.
Post by Вадим Яницкий
Hi folks!
I'm new in A5/1 world and I have read that 2 Tb is not enough for table
conversation.
Post by Вадим Яницкий
But I have a server with 2 Tb and 60 Gb of additional space only.
My question is: is it possible to convert the tables one by one? I.e.
can I download first table (about 41 Gb) on my 60 Gb HDD and convert it,
download next one and convert with offset ...?
Post by Вадим Яницкий
And one more question. Can anyone explain me the Behemoth.py
configuration format? What means this section: "#Tables: dev id(advance)
offset"? Maybe it is an answer to my previous question. :)
Post by Вадим Яницкий
With best regards,
Vadim Yanitskiy.
_______________________________________________
A51 mailing list
https://lists.srlabs.de/cgi-bin/mailman/listinfo/a51
--
Jan Hrach | http://jenda.hrach.eu/
GPG CD98 5440 4372 0C6D 164D A24D F019 2F8E 6527 282E
Jan Hrach
2016-01-18 17:14:19 UTC
Permalink
Raw Message
1-3.1 seem correct
3.2) In case of RAW storage-method there is no file system and I have to calculate offset for every new table. How can I do that?
TableConvert tells you how many bytes were written (and if you are converting to files, you will see this as a file size). I'd round this number up to the nearest multiply of 4096 bytes and use this as an offset for the next one.

I don't know how much performance penalty is imposed by having tables in files instead of the raw device. I hope not so much and I'd leave it in files at least during experimenting.
Thanks for explanations!
Though I'd better keep the original files in case you find out something is screwed up.
It may sound sad, but I have no more than 2 Tb of space.
I have no even direct HDD access because my server is virtualized and running on the Hyper-V (it's not my own).
1) Prepare 2 Tb HDD changing the ownership using `chown`.
1.1) In case of file system storage-method format it to ext4, for example.
2) Download the first part (a51_table_100.dlt) to the home folder
of my "buffer-hdd" (60 Gb).
./TableConvert di (/dev/sda)/home/$file.dlt \
(/dev/sdb)/$file.ins:0 (/dev/sda)/home/kraken/indexes/$file.idx
on every downloaded file. And in this case the offset param will always = 0.
3.2) In case of RAW storage-method there is no file system and I have to calculate offset for every new table. How can I do that?
Is this sequence of actionsright?
With best regards,
Vadim Yanitsky.
I'm new in A5/1 world and I have read that 2 Tb is not enough for table conversation.
It is (you can convert it one-by-one, requiring only 40 additional GB). Though I'd better keep the original files in case you find out something is screwed up.
See https://brmlab.cz/project/gsm/deka/deka-admin
Can anyone explain me the Behemoth.py configuration format? What means this section: "#Tables: dev id(advance) offset"?
The hash-mark lines are the comments.
I think you want to tell it somehow how many tables should be installed on which device, but that's not the case with one-by-one installation.
If you have it on filesystem as files, I'd just run
./TableConvert di $file.dlt $file.ins:0 $file.idx
on every file. You can do the block device installation later with dd seek=X.
Hi folks!
I'm new in A5/1 world and I have read that 2 Tb is not enough for table conversation.
But I have a server with 2 Tb and 60 Gb of additional space only.
My question is: is it possible to convert the tables one by one? I.e. can I download first table (about 41 Gb) on my 60 Gb HDD and convert it, download next one and convert with offset ...?
And one more question. Can anyone explain me the Behemoth.py configuration format? What means this section: "#Tables: dev id(advance) offset"? Maybe it is an answer to my previous question. :)
With best regards,
Vadim Yanitskiy.
_______________________________________________
A51 mailing list
https://lists.srlabs.de/cgi-bin/mailman/listinfo/a51
--
Jan Hrach | http://jenda.hrach.eu/
GPG CD98 5440 4372 0C6D 164D A24D F019 2F8E 6527 282E
_______________________________________________
A51 mailing list
https://lists.srlabs.de/cgi-bin/mailman/listinfo/a51
--
Jan Hrach | http://jenda.hrach.eu/
GPG CD98 5440 4372 0C6D 164D A24D F019 2F8E 6527 282E
Loading...