[memfile] Add support for faking files in memory - [merged] #158

Closed
opened 2020-10-18 23:50:49 +03:00 by icex2 · 50 comments
icex2 commented 2020-10-18 23:50:49 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 22:50

Merges memfile -> master

Also fix iohook race condition

In GitLab by @xyen on Oct 18, 2020, 22:50 _Merges memfile -> master_ Also fix iohook race condition
icex2 commented 2020-10-18 23:55:00 +03:00 (Migrated from github.com)

If you already have enum memfile_hook_path_mode, which is good, then I suggest using that type here in the parameters instead of the int32_t to clarify on the values available.

If you already have `enum memfile_hook_path_mode`, which is good, then I suggest using that type here in the parameters instead of the `int32_t` to clarify on the values available.
icex2 commented 2020-10-18 23:55:57 +03:00 (Migrated from github.com)

Isn't there a MAX_PATH define for windows? If there are no further constraints for using 256 max, I suggest using the maximum defined by windows.

Isn't there a `MAX_PATH` define for windows? If there are no further constraints for using 256 max, I suggest using the maximum defined by windows.
icex2 commented 2020-10-18 23:57:01 +03:00 (Migrated from github.com)

Can you provide a brief comment about what this whole module is about, kinda like "This is initializing the module for use. You can do this and that with it".

Can you provide a brief comment about what this whole module is about, kinda like "This is initializing the module for use. You can do this and that with it".
icex2 commented 2020-10-18 23:57:40 +03:00 (Migrated from github.com)

Nit: Not really a valuable comment, at least here. Can be removed.

Nit: Not really a valuable comment, at least here. Can be removed.
icex2 commented 2020-10-18 23:58:50 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 22:58

Commented on src/main/hooklib/memfile.h line 27

changed this line in version 2 of the diff

In GitLab by @xyen on Oct 18, 2020, 22:58 Commented on [src/main/hooklib/memfile.h line 27](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..cb18d35c00cb1488b47fc7bd439a60dbad56323d#diff-c224cd3033f3297cd47c8f1e895d7e9fR27) changed this line in [version 2 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1340&start_sha=cb18d35c00cb1488b47fc7bd439a60dbad56323d#04193e1f22b80fa16b3dde54255dd293059f1509_27_27)
icex2 commented 2020-10-18 23:58:50 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 22:58

Commented on src/main/hooklib/memfile.c line 14

changed this line in version 2 of the diff

In GitLab by @xyen on Oct 18, 2020, 22:58 Commented on [src/main/hooklib/memfile.c line 14](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..cb18d35c00cb1488b47fc7bd439a60dbad56323d#diff-fd6ce73b76a2d6daf2daf9af42786a0cR14) changed this line in [version 2 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1340&start_sha=cb18d35c00cb1488b47fc7bd439a60dbad56323d#0c57b152ed0ac0009821113726a2bef5a0c468f3_14_14)
icex2 commented 2020-10-18 23:58:51 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 22:58

added 1 commit

  • c45a05aa - [memfile] Add support for faking files in memory

Compare with previous version

In GitLab by @xyen on Oct 18, 2020, 22:58 added 1 commit <ul><li>c45a05aa - [memfile] Add support for faking files in memory</li></ul> [Compare with previous version](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1340&start_sha=cb18d35c00cb1488b47fc7bd439a60dbad56323d)
icex2 commented 2020-10-19 00:00:09 +03:00 (Migrated from github.com)

For tracability, I suggest adding a log_info("Initialized") at the end of the function.

For tracability, I suggest adding a `log_info("Initialized")` at the end of the function.
icex2 commented 2020-10-19 00:00:36 +03:00 (Migrated from github.com)

Nit: Can be shortened to log_info("Finished") because the logging will include module names

Nit: Can be shortened to `log_info("Finished")` because the logging will include module names
icex2 commented 2020-10-19 00:02:38 +03:00 (Migrated from github.com)

I am wondering about the convenience of making the caller having to maintain the memory that is used as data. What do you think? Wouldn't it be more convenient for the caller to just dynamically allocate something, pass it to the memfile backend and let it clean up when the backend is cleaned up?

I am wondering about the convenience of making the caller having to maintain the memory that is used as data. What do you think? Wouldn't it be more convenient for the caller to just dynamically allocate something, pass it to the memfile backend and let it clean up when the backend is cleaned up?
icex2 commented 2020-10-19 00:04:14 +03:00 (Migrated from github.com)

For tracability (again), I would add a log_misc("Add %s, mode %d, data size %d", path, path_mode, sz) call here.

For tracability (again), I would add a `log_misc("Add %s, mode %d, data size %d", path, path_mode, sz)` call here.
icex2 commented 2020-10-19 00:06:23 +03:00 (Migrated from github.com)

Nit: Add log_assert(path), log_assert(data)

Nit: Add `log_assert(path)`, `log_assert(data)`
icex2 commented 2020-10-19 00:07:21 +03:00 (Migrated from github.com)

Nit: Line break before control block

Nit: Line break before control block
icex2 commented 2020-10-19 00:08:04 +03:00 (Migrated from github.com)

I think this message might be too verbose for log_info as a log level and log_misc suits it better.

I think this message might be too verbose for `log_info` as a log level and `log_misc` suits it better.
icex2 commented 2020-10-19 00:08:12 +03:00 (Migrated from github.com)

Same here

Same here
icex2 commented 2020-10-19 00:08:41 +03:00 (Migrated from github.com)

Nit: Line break after this and before control block

Nit: Line break after this and before control block
icex2 commented 2020-10-19 00:09:03 +03:00 (Migrated from github.com)

Nit: Add log_assert(entry) and log_assert(irp)

Nit: Add `log_assert(entry)` and `log_assert(irp)`
icex2 commented 2020-10-19 00:09:32 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:09

Commented on src/main/hooklib/memfile.h line 35

If you pass it static strings like:

memfile_hook_add_fd("d:\\\\somefile.txt", ABSOLUTE_MATCH, "ABC", 3);

It'll exist for the lifetime of the application. Additionally in the future, if we add write support, having the buffer be caller controlled, lets them introspect / use the written contents.

In GitLab by @xyen on Oct 18, 2020, 23:09 Commented on [src/main/hooklib/memfile.h line 35](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..62ebe3108332464b48e9989da0d37279583e64f3#diff-c224cd3033f3297cd47c8f1e895d7e9fR35) If you pass it static strings like: `memfile_hook_add_fd("d:\\\\somefile.txt", ABSOLUTE_MATCH, "ABC", 3);` It'll exist for the lifetime of the application. Additionally in the future, if we add write support, having the buffer be caller controlled, lets them introspect / use the written contents.
icex2 commented 2020-10-19 00:10:51 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:10

Commented on src/main/hooklib/memfile.c line 90

data can be null, there's cases (such as sdvx) that check for file existence only. will add assert for path

In GitLab by @xyen on Oct 18, 2020, 23:10 Commented on [src/main/hooklib/memfile.c line 90](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..62ebe3108332464b48e9989da0d37279583e64f3#diff-fd6ce73b76a2d6daf2daf9af42786a0cR90) data can be null, there's cases (such as sdvx) that check for file existence only. will add assert for path
icex2 commented 2020-10-19 00:10:55 +03:00 (Migrated from github.com)

Shouldn't this include the current position of both, the in-memory buffer and the irp read buffer? Otherwise, this might fail on chunk'd reads.

        memcpy(irp->read.bytes + irp->read.pos, entry->data + entry->pos, nread);
Shouldn't this include the current position of both, the in-memory buffer and the irp read buffer? Otherwise, this might fail on chunk'd reads. ```suggestion:-0+0 memcpy(irp->read.bytes + irp->read.pos, entry->data + entry->pos, nread); ```
icex2 commented 2020-10-19 00:11:15 +03:00 (Migrated from github.com)

Add log_assert checks for params again

Add `log_assert` checks for params again
icex2 commented 2020-10-19 00:12:23 +03:00 (Migrated from github.com)

Nit: For the very unlikely case, I would add a else branch with a log_die("Illegal state") here. Such things can be super painful to track down if they happen.

Nit: For the very unlikely case, I would add a else branch with a `log_die("Illegal state")` here. Such things can be super painful to track down if they happen.
icex2 commented 2020-10-19 00:12:41 +03:00 (Migrated from github.com)

More log_assert of params

More `log_assert` of params
icex2 commented 2020-10-19 00:13:34 +03:00 (Migrated from github.com)

Nit: This could be turned into a switch statement. Might increase readability a bit.

Nit: This could be turned into a switch statement. Might increase readability a bit.
icex2 commented 2020-10-19 00:14:57 +03:00 (Migrated from github.com)

I have a feeling that this is not always going to work. Wouldn't it make sense to just tell the application everything's fine instead? Since you keep the file cached as long as process is alive, what happens if a file gets re-opened at some point?

I have a feeling that this is not always going to work. Wouldn't it make sense to just tell the application everything's fine instead? Since you keep the file cached as long as process is alive, what happens if a file gets re-opened at some point?
icex2 commented 2020-10-19 00:15:59 +03:00 (Migrated from github.com)

Since this could still be called on a file and someone might be wondering why things fail, if this even surfaces at all, I suggest adding a log_warn("Write unsupported") here.

Same with the other functions that are not supported: ioctl, fsync.

Since this could still be called on a file and someone might be wondering why things fail, if this even surfaces at all, I suggest adding a `log_warn("Write unsupported")` here. Same with the other functions that are not supported: ioctl, fsync.
icex2 commented 2020-10-19 00:16:47 +03:00 (Migrated from github.com)

Why do you return false here? I would consider this as S_OK as the error will be delivered to the application and user.

Why do you return false here? I would consider this as `S_OK` as the error will be delivered to the application and user.
icex2 commented 2020-10-19 00:20:57 +03:00 (Migrated from github.com)

Nit: Line break before control block. Also applies two lines after this one.

Nit: Line break before control block. Also applies two lines after this one.
icex2 commented 2020-10-19 00:22:06 +03:00 (Migrated from github.com)

For clarity, I would add a brief comment, maybe on the doc of memfile_hook_add_fd in the header file that you are currently supporting readonly, only.

For clarity, I would add a brief comment, maybe on the doc of `memfile_hook_add_fd` in the header file that you are currently supporting readonly, only.
icex2 commented 2020-10-19 00:23:54 +03:00 (Migrated from github.com)

Why and when would you want to use this option?

Why and when would you want to use this option?
icex2 commented 2020-10-19 00:24:16 +03:00 (Migrated from github.com)

Since this is a very important bugfix, I highly advice to have this at least in a separate commit.

Since this is a very important bugfix, I highly advice to have this at least in a separate commit.
icex2 commented 2020-10-19 00:28:45 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:28

Commented on dist/sdvx5/sdvxhook2.conf line 13

In theory every iohook IRP we add introduces additional overhead, it's also useful in case this breaks something that we're unaware of.

In GitLab by @xyen on Oct 18, 2020, 23:28 Commented on [dist/sdvx5/sdvxhook2.conf line 13](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..62ebe3108332464b48e9989da0d37279583e64f3#diff-508abdbaee1a5f897c3fe1d4a9cf1543R13) In theory every iohook IRP we add introduces additional overhead, it's also useful in case this breaks something that we're unaware of.
icex2 commented 2020-10-19 00:29:39 +03:00 (Migrated from github.com)

Nit: Code style, line break before control block

Nit: Code style, line break before control block
icex2 commented 2020-10-19 00:30:59 +03:00 (Migrated from github.com)

I wouldn't log this outside of memfile_hook_add_fd if that call covers a more generic "you just added this hook fd". -> Remove log call here

I wouldn't log this outside of `memfile_hook_add_fd` if that call covers a more generic "you just added this hook fd". -> Remove log call here
icex2 commented 2020-10-19 00:32:15 +03:00 (Migrated from github.com)

Remove this. Logging in memfile_hook module should already cover tracabilty.

Remove this. Logging in memfile_hook module should already cover tracabilty.
icex2 commented 2020-10-19 00:46:55 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 46

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 46](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR46) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_46_46)
icex2 commented 2020-10-19 00:46:55 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 80

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 80](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR80) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_80_81)
icex2 commented 2020-10-19 00:46:56 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 121

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 121](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR121) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_121_127)
icex2 commented 2020-10-19 00:46:56 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 126

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 126](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR126) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_126_132)
icex2 commented 2020-10-19 00:46:56 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 147

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 147](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR147) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_147_158)
icex2 commented 2020-10-19 00:46:56 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 158

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 158](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR158) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_158_169)
icex2 commented 2020-10-19 00:46:57 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 180

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 180](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR180) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_180_203)
icex2 commented 2020-10-19 00:46:57 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 186

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 186](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR186) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_186_209)
icex2 commented 2020-10-19 00:46:57 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/hooklib/memfile.c line 177

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/hooklib/memfile.c line 177](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-fd6ce73b76a2d6daf2daf9af42786a0cR177) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#0c57b152ed0ac0009821113726a2bef5a0c468f3_177_200)
icex2 commented 2020-10-19 00:46:58 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/sdvxhook2/dllmain.c line 64

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/sdvxhook2/dllmain.c line 64](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-ec3dc1d5d05e6c2d84f62d40fe3ec1e8R64) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#b617b3e69277bb24c278ccb4f3afaca1472ab912_64_65)
icex2 commented 2020-10-19 00:46:58 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

Commented on src/main/sdvxhook2/dllmain.c line 137

changed this line in version 3 of the diff

In GitLab by @xyen on Oct 18, 2020, 23:46 Commented on [src/main/sdvxhook2/dllmain.c line 137](https://github.com/djhackersdev/bemanitools/compare/ed413aad08a372f6c03ac44fec25695641a212e2..c45a05aa636d45672a4089df883ad7724c283c51#diff-ec3dc1d5d05e6c2d84f62d40fe3ec1e8R137) changed this line in [version 3 of the diff](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51#b617b3e69277bb24c278ccb4f3afaca1472ab912_137_136)
icex2 commented 2020-10-19 00:46:58 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:46

added 2 commits

  • 998fdafa - [iohook] Fix iohook_init race condition
  • 62ebe310 - [memfile] Add support for faking files in memory

Compare with previous version

In GitLab by @xyen on Oct 18, 2020, 23:46 added 2 commits <ul><li>998fdafa - [iohook] Fix iohook_init race condition</li><li>62ebe310 - [memfile] Add support for faking files in memory</li></ul> [Compare with previous version](/djhackers/bemanitools/-/merge_requests/57/diffs?diff_id=1341&start_sha=c45a05aa636d45672a4089df883ad7724c283c51)
icex2 commented 2020-10-19 00:49:20 +03:00 (Migrated from github.com)

In GitLab by @xyen on Oct 18, 2020, 23:49

resolved all threads

In GitLab by @xyen on Oct 18, 2020, 23:49 resolved all threads
icex2 commented 2020-10-19 00:55:43 +03:00 (Migrated from github.com)

Nice work, lgtm.

Nice work, lgtm.
icex2 commented 2020-10-19 00:55:46 +03:00 (Migrated from github.com)

approved this merge request

approved this merge request
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Max/djhackersdev_bemanitools#158