Arch Linux User Repository

I apologize, I falsely assumed that since the service puppetserver.service was run with User=puppet, that puppetserver only cared about the user permissions on a particular file. Apparently, that assumption is inaccurate, as making the change @lb.laboon suggested (chown --recursive puppet:puppet /etc/puppetlabs/puppetserver/ca/) has resolved my issue. I'm not sure why it suddenly changed, but I'll take the win.

@trainzkid I don't use the CA functionality personally so I'm probably not the best person to ask, but it sounds like you might wanna make sure everything under /etc/puppetlabs/puppetserver/ca has puppet:puppet ownership.

I spoke a little too soon I guess. After the --rebuild, I had run sudo puppetserver ca, as I had tried to do prior, and it had worked so I thought my issue was resolved.

Now while trying to sign a certificate for a new node/computer, it's giving me an error. I've not seen this error before, so if it's something obvious, please let me know:

$USER@$HOST ~> sudo puppetserver ca sign --certname $CERT_NAME
Error:
    When attempting to sign certificate request '$CERT_NAME', received
      code: 500
      body: Internal Server Error: java.nio.file.FileSystemException: /etc/puppetlabs/puppetserver/ca/inventory.txt13130874269473365363tmp: Operation not permitted

The file in question seems to have the correct permissions:

$USER@$HOST ~> sudo exa --long --group /etc/puppetlabs/puppetserver/ca/ | grep inventory
.rw-rw----    0 puppet root    6 Mar  2023 infra_inventory.txt
.rw-rw----  469 puppet root   15 Nov  2023 inventory.txt
.rw-------  538 puppet puppet 19 Aug 22:06 inventory.txt3676345110244129600tmp
.rw-------  545 puppet puppet 19 Aug 22:21 inventory.txt9844678833141241189tmp
.rw-------  545 puppet puppet 19 Aug 22:09 inventory.txt13130874269473365363tmp
.rw-------  545 puppet puppet 19 Aug 22:06 inventory.txt13902248159591484979tmp

This help page seems to suggest that this error is related to a permissions issue on /etc/puppetlabs/puppet/ssl/ca/serial, but I'm not seeing that issue on my server:

$USER@$HOST ~ [0|1]> sudo exa --long --group /etc/puppetlabs/puppet/ssl/ca | grep serial
.rw-rw-r--    0 puppet root    6 Mar  2023 infra_serials
.rw-------    0 puppet puppet 12 Apr  2023 infra_serials526946893511203747tmp
.rw--w----    0 puppet root    6 Mar  2023 infra_serials1039029428571312054tmp
.rw--w----    0 puppet puppet  7 Mar  2023 infra_serials3198766218664331873tmp
.rw--w----    0 puppet root    6 Mar  2023 infra_serials5926309510903958816tmp
.rw-r--r--    4 puppet puppet 19 Aug 22:45 serial

A little paru --rebuild puppetserver and a service restart and I'm back in business!

Thanks again, @lb.laboon!!

This package needs a rebuild for the latest ruby (didn't realize that on my own stuff because I was rebuilding for other reasons). I just bumped the pkgrel for that.

I hate to come back with more issues, but I've just discovered that puppetserver ca commands don't seem to work, I get an error:

$USER@$HOST ~ [1]> sudo puppetserver ca
<internal:/usr/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- puppetserver/ca/cli (LoadError)
        from <internal:/usr/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
        from /opt/puppetlabs/server/apps/puppetserver/cli/apps/ca:3:in `<main>'

Another config option maybe?

Updated the following line in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf from:

ruby-load-path: [/usr/lib/ruby/vendor_ruby/3.0.0,/usr/lib/ruby/vendor_ruby]                                          

to:

ruby-load-path: [/usr/lib/ruby/vendor_ruby/3.2.0,/usr/lib/ruby/vendor_ruby]

puppetserver seems to be running happily now (albeit with a few complaints), thanks @lb.laboon! Logs after change for reference:

Starting puppetserver Service...
WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
15:53:43,928 |-INFO in LogbackRequestLog - Will use configuration file [/etc/puppetlabs/puppetserver/request-logging.xml]
15:53:43,931 |-INFO in ch.qos.logback.access.model.processor.ConfigurationModelHandler - debug attribute not set
15:53:43,931 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [FILE]
15:53:43,931 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
15:53:43,932 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1271866328 - setting totalSizeCap to 1 GB
15:53:43,932 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1271866328 - Archive files will be limited to [200 MB] each.
15:53:43,932 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1271866328 - Will use gz compression
15:53:43,932 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1271866328 - Will use the pattern /var/log/puppetlabs/puppetserver/puppetserver-access-%d{yyyy-MM-dd}.%i.log for the active file
15:53:43,932 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@62c80a31 - The date pattern is 'yyyy-MM-dd' from file name pattern '/var/log/puppetlabs/puppetserver/puppetserver-access-%d{yyyy-MM-dd}.%i.log.gz'.
15:53:43,932 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@62c80a31 - Roll-over at midnight.
15:53:43,932 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@62c80a31 - Setting initial period to 2024-08-16T05:32:30.201Z
15:53:43,933 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
15:53:43,934 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /var/log/puppetlabs/puppetserver/puppetserver-access.log
15:53:43,934 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/var/log/puppetlabs/puppetserver/puppetserver-access.log]
15:53:43,934 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [FILE] to null
15:53:43,934 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@6d5a91fd - End of configuration.
15:53:43,934 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@51ec954 - Registering current configuration as safe fallback point
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Started puppetserver Service.
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f

@trainzkid Check the ruby-load-path option in your puppetserver.conf. It will need to be updated for the new version of ruby that Arch just shipped.

Unrelated, but in case anybody was wondering this package is a couple versions out of date and I am leaving it at the current version for the time being because there are some issues with the latest versions that (I think) are a result of the very outdated version of facter that Arch ships in the official repos. I'm experimenting around a bit with newer versions of facter, puppet, etc, unsuccessfully so far but if anybody finds something that works let me know!

I'm having trouble getting this application to run following a recent system update, which appears to have changed a bunch of ruby packages:

Starting puppetserver Service...
WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs
Execution error (LoadError) at org.jruby.RubyKernel/require (org/jruby/RubyKernel.java:1071).
(LoadError) no such file to load -- puppet
Full report at:
/tmp/clojure-8639881353200068708.edn
Background process 1168 exited before start had completed
puppetserver.service: Control process exited, code=exited, status=1/FAILURE

Doesn't seem to work on a fresh install (in an nspawn container) either:

/usr/lib/systemd/system/puppetserver.service:45: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Starting puppetserver Service...
WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Execution error (RuntimeError) at RUBY/use (/usr/lib/ruby/vendor_ruby/3.2.0/puppet/settings.rb:1152).
(RuntimeError) Got 1 failure(s) while initializing: File[/etc/puppetlabs/puppet/ssl]: change from 'absent' to 'directory' failed: Could not set 'directory' on ensure: Permission denied - /etc/puppetlabs/puppet/ssl
Full report at:
/tmp/clojure-5623376312178173036.edn
Background process 27031 exited before start had completed
puppetserver.service: Control process exited, code=exited, status=1/FAILURE

The system update indicates a bunch of ruby packages were merged into extra/ruby (as I understand it?):

...
:: Starting full system upgrade...
:: Replace ruby-abbrev with extra/ruby? [Y/n]
:: Replace ruby-base64 with extra/ruby? [Y/n]
:: Replace ruby-benchmark with extra/ruby? [Y/n]
:: Replace ruby-bigdecimal with extra/ruby? [Y/n]
:: Replace ruby-bundledgems with extra/ruby-bundled-gems? [Y/n]
:: Replace ruby-cgi with extra/ruby? [Y/n]
:: Replace ruby-csv with extra/ruby? [Y/n]
:: Replace ruby-date with extra/ruby? [Y/n]
:: Replace ruby-delegate with extra/ruby? [Y/n]
:: Replace ruby-did_you_mean with extra/ruby? [Y/n]
:: Replace ruby-digest with extra/ruby? [Y/n]
:: Replace ruby-drb with extra/ruby? [Y/n]
:: Replace ruby-english with extra/ruby? [Y/n]
:: Replace ruby-etc with extra/ruby? [Y/n]
:: Replace ruby-fcntl with extra/ruby? [Y/n]
:: Replace ruby-fiddle with extra/ruby? [Y/n]
:: Replace ruby-fileutils with extra/ruby? [Y/n]
:: Replace ruby-find with extra/ruby? [Y/n]
:: Replace ruby-forwardable with extra/ruby? [Y/n]
:: Replace ruby-getoptlong with extra/ruby? [Y/n]
:: Replace ruby-io-console with extra/ruby? [Y/n]
:: Replace ruby-io-nonblock with extra/ruby? [Y/n]
:: Replace ruby-io-wait with extra/ruby? [Y/n]
:: Replace ruby-ipaddr with extra/ruby? [Y/n]
:: Replace ruby-json with extra/ruby? [Y/n]
:: Replace ruby-logger with extra/ruby? [Y/n]
:: Replace ruby-mutex_m with extra/ruby? [Y/n]
:: Replace ruby-net-http with extra/ruby? [Y/n]
:: Replace ruby-open-uri with extra/ruby? [Y/n]
:: Replace ruby-psych with extra/ruby? [Y/n]
:: Replace ruby-reline with extra/ruby? [Y/n]
:: Replace ruby-ruby2_keywords with extra/ruby? [Y/n]
:: Replace ruby-stringio with extra/ruby? [Y/n]
:: Replace ruby-time with extra/ruby? [Y/n]
:: Replace ruby-tmpdir with extra/ruby? [Y/n]
:: Replace ruby-uri with extra/ruby? [Y/n]
resolving dependencies...
...

Let me know if there's something stupid/obvious I'm missing!

Really dumb mistake I made that others may make, so just for visibility!!!! If puppetserver fails to start because it can't find java, make sure your default Java environment is set correctly.