Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

agnos.is Forums

  1. Home
  2. Technical Support
  3. How to efficiently import ~16M users with custom fields?

How to efficiently import ~16M users with custom fields?

Scheduled Pinned Locked Moved Technical Support
4 Posts 2 Posters 0 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • anton-sobchuk@community.nodebb.orgA This user is from outside of this forum
    anton-sobchuk@community.nodebb.orgA This user is from outside of this forum
    [email protected]
    wrote last edited by
    #1

    Hi everyone,

    I’m working on a project where I need to import approximately 16 million users into a NodeBB forum.
    In addition to the standard fields, I also need to set some custom fields for each user during the import.

    I’ve already tried a few approaches:

    • Direct writes to MongoDB (objects collection), but it seems there’s a lot happening under the hood when creating a user (hooks, hashes, plugins, etc.), and it’s almost impossible to replicate everything manually.
    • Using the Write API (POST /api/v3/users) works correctly, but it’s painfully slow at scale.
    • The CSV Import plugin doesn’t allow me to set the user’s creation date (timestamp), which is essential for my use case, so it’s not an option either.

    Has anyone dealt with importing a very large number of users into NodeBB?
    Do you have any recommendations or best practices for this kind of task?

    Thanks in advance for any advice or ideas!

    baris@community.nodebb.orgB 1 Reply Last reply
    0
    • anton-sobchuk@community.nodebb.orgA [email protected]

      Hi everyone,

      I’m working on a project where I need to import approximately 16 million users into a NodeBB forum.
      In addition to the standard fields, I also need to set some custom fields for each user during the import.

      I’ve already tried a few approaches:

      • Direct writes to MongoDB (objects collection), but it seems there’s a lot happening under the hood when creating a user (hooks, hashes, plugins, etc.), and it’s almost impossible to replicate everything manually.
      • Using the Write API (POST /api/v3/users) works correctly, but it’s painfully slow at scale.
      • The CSV Import plugin doesn’t allow me to set the user’s creation date (timestamp), which is essential for my use case, so it’s not an option either.

      Has anyone dealt with importing a very large number of users into NodeBB?
      Do you have any recommendations or best practices for this kind of task?

      Thanks in advance for any advice or ideas!

      baris@community.nodebb.orgB This user is from outside of this forum
      baris@community.nodebb.orgB This user is from outside of this forum
      [email protected]
      wrote last edited by [email protected]
      #2

      Somethings you can try:

      • Bypass the write api and call User.create directly in a migration script.
      • One of the expensive parts of User.create is hashing the password, you can disable this part and force everyone to reset their password.
      • Disable sending a welcome notification everytime a user is created. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L101)]
      • Disable welcome email. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L111-L117)]
      • Disable registration analytics. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L98)]
      • If all the usernames are unique in your data you can disable the unique name generator. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L65-L70)]
      • Same as above but for emails, if all the emails are already unique you can disable the email available check. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L152-L157)]
      • Comment out the locking code for username/email[[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L24-L27)]

      These are just some of the low effort ones.

      1 Reply Last reply
      1
      • anton-sobchuk@community.nodebb.orgA This user is from outside of this forum
        anton-sobchuk@community.nodebb.orgA This user is from outside of this forum
        [email protected]
        wrote last edited by
        #3

        baris Thank you for the suggestions, much appreciated! 🙌

        By the way, is there any built-in or recommended mechanism to transfer the entire database to another NodeBB instance? Or is it just a matter of dumping and restoring the database manually?

        1 Reply Last reply
        0
        • baris@community.nodebb.orgB This user is from outside of this forum
          baris@community.nodebb.orgB This user is from outside of this forum
          [email protected]
          wrote last edited by
          #4

          Yes tranfering the database is just a mongodump and mongorestore. If you want the uploads as well copy the public/uploads folder.

          1 Reply Last reply
          1
          Reply
          • Reply as topic
          Log in to reply
          • Oldest to Newest
          • Newest to Oldest
          • Most Votes


          • Login

          • Login or register to search.
          • First post
            Last post
          0
          • Categories
          • Recent
          • Tags
          • Popular
          • World
          • Users
          • Groups