Skip to content

Comments

Fix conversion of Uint8Array to object (#761)#762

Open
Maia-Everett wants to merge 1 commit intonode-config:masterfrom
Maia-Everett:uint8array
Open

Fix conversion of Uint8Array to object (#761)#762
Maia-Everett wants to merge 1 commit intonode-config:masterfrom
Maia-Everett:uint8array

Conversation

@Maia-Everett
Copy link

As said in #761, node-config handles TypedArray values (including Uint8Array) incorrectly, wrapping them in proxy objects that are not accepted by anything that expects a real TypedArray (e.g. Buffer.from()).

I hit this bug with js-yaml 4.x, which made a breaking change compared to js-yaml 3.x, returning Uint8Array for !!binary values rather than returning Buffer like 3.x did.

This PR:

  • Makes makeImmutable() ignore TypedArray and DataView values (leaving them mutable, as there's no way to make them immutable)
  • Makes cloneDeep() correctly clone TypedArray and DataView values
  • Bumps the js-yaml dev dependency to the latest version 4.1.0, which hits the bug in node-config
  • Adds a test for the bug fixed

Copy link
Collaborator

@markstos markstos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@markstos
Copy link
Collaborator

@lorenwest This PR seems consistent with some other special cases we handle and includes test coverage. I recommend merging it.

@markstos
Copy link
Collaborator

markstos commented Jan 7, 2025

A competing PR was merged, but there are parts I like better here. I like that this returns Uint8Array and not a buffer, and I think the test coverage here was more clearly organized. I would like to get the test coverage contributed for both cases merged like this:

'use strict';

const requireUncached = require('./_utils/requireUncached');
require('../parser');

'use strict';

const vows = require('vows');
const assert = require('assert');

vows.describe('Tests for parsing binary data')
  .addBatch({
    'Using the YAML parser - Uint8Array': {
      topic: function() {
        process.env.NODE_CONFIG_DIR = __dirname + '/22-binary';
        return requireUncached(__dirname + '/../lib/config');
      },
      'A binary value should be immutable': function(config) {
        assert.throws(() => {
          const auth = config.get('auth');
          auth.secret = new Uint8Array([ 1, 2, 3 ]);
        }, /Can not update runtime configuration/)
      },

      'reading !!binary returns a real Uint8Array': function(CONFIG) {
        assert.deepStrictEqual(CONFIG.get('auth'), {
					secret: new Uint8Array([0x10, 0x11, 0x12, 0x13, 0x14, 0x15])
				});
      },
    }
  })
  .export(module);

@jdmarshall
Copy link
Collaborator

jdmarshall commented Apr 16, 2025

I don’t think exposing mutable data was meant to be part of the Config design and API.

In fact I’m not happy with the current Buffer behavior referenced at the top of this function. That seems wrong. It should at least be a copy operation.

I believe if you’ve disabled immutability this code doesn’t get called at all, so we are just talking about the default behavior of disallowing modification after initialization.

Mutability make the config into global shared state. There’s a place for that, and if you can overlay a single writer api over the top of node-config you can do a lot of cool stuff. As long as every function you call can modify that state it’s chaos for local reasoning. Not to mention testing. You might as well just use global.*.

Global shared state doesn’t scale past a few ambitious coworkers trying to “get things done” and deal with consequences “later”. If you’ve picked up node-config you probably trying to avoid that

I think I’d be open to snapshotting. And a new TypedArray doesn’t have to be built on a Buffer. It can be created on an Array built from the source buffer. Though how are you even getting a buffer into config in the first place?

@markstos
Copy link
Collaborator

Ancient versions of the library did support mutability. Making the config structure immutable was something I advocated and Loren agreed, and keeping the config state immutable has been a priority since.

@jdmarshall
Copy link
Collaborator

Good, then we are all three in accord (I've had this conversation before with Loren as well).

I think there is a hole here, that I would like to make smaller or close if we can. But I think this solution is going left when we want to go right.

An part of that is knowing more about the use case. I'm still in the 'why would you encounter this?' bubble and that's not a good place to make a counterproposal from.

@markstos
Copy link
Collaborator

I'm still in the 'why would you encounter this?' bubble and that's not a good place to make a counterproposal from.

There's a test case to illustrate. It seems it can happen when you store binary objects in a YAML config file.

@markstos
Copy link
Collaborator

markstos commented Jun 3, 2025

It looks like there's alignment about merging this, but now there are conflicts that need to be resolved.

@jdmarshall
Copy link
Collaborator

I don't know if I'd say there's alignment but I am currently bumping into this issue trying to clean up makeImmutable, which is only guarding against converting Buffers at the top level but not when looking at the children.

But again, why do we have features that were supported to add a mutable object into the middle of immutable data?

return object;
}

if (util.isTypedArray(object)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to be clear that I'm not trying to shoot the messenger here.

This function is a mess. And every case we add to it makes it more obvious to me that it's a mess.

It's a bad copy of cloneDeep, and I'm encountering places where we are recursing objects we should not be recursing. I think this needs a rethink. I'm trying to write a benchmark loop right now to make sure I'm not introducing regressions, and it's breaking because somehow we've gotten past the Buffer guard at the top of the function and now I'm getting errors from the TypedArray inside the Buffer. So there's that.

I just discovered the other day that the reason makeImmutable() couldn't be moved to Util is actually false coupling. So I think it can be moved, leaving the one, intrusive backward compatibility feature in config.util (and deprecating it) and rearrange this mess to look more like cloneDeep, which it should in fact be a sibling to and so very much isn't. Probably because different people have cared about different bugs in each.

I have enough open PRs at the moment, with two more in the chamber, so I won't be tackling that right now but for sure that's on the roadmap now.

@jdmarshall
Copy link
Collaborator

So here's what I propose.

I'm going to take on extracting makeImmutable to Util for 4.4 or a 4.5 build.

I'm going to pull in some of @Maia-Everett's commits to retain attribution, then I'm going to move the code to lib/util.js, rearrange some of the guts of makeImmutable and cloneDeep so they don't look so much like individual snowflakes. I think I prefer the shape of cloneDeep to makeImmutable. I might need to extract some bits to make that happen.

Then I'm going to deprecate that config.util.makeImmutable(obj, propertyName, value) feature because what is that even for? It's making a method that is getting called a massive number of times on large config bodies that much slower and hard to read. And we don't even use it, and haven't for most of the life of this project. GTFO.

But first I have too many branches open at the same time and I need to land some of those before I can tackle this.

@jdmarshall jdmarshall added this to the 4.4 milestone Jan 20, 2026
@jdmarshall
Copy link
Collaborator

There's already a change earmarked for 4.4 that affects how makeImmutable works. Might as well make it two to reduce the number of fire drills users have to experience.

@jdmarshall
Copy link
Collaborator

I'm having second thoughts on this one. Without these changes, it seems that you can get what you want by using

Buffer.from(config.auth.secret);

@jdmarshall jdmarshall removed this from the 4.4 milestone Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants