September 2021 Editon of The Resource
Hello Reader, here is this month’s iRODS news and developments!
If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or drop me a mail to say ‘Hi’. Always lovely to hear from people, particularly in these pandemic times!
I’d love your thoughts and feedback on how this newsletter could be better for you.
News
iRODS Web GUI - Metalnx 2.5.0 is released.
The refactored search is now a single tab, only uses GenQuery, and therefore better honors iRODS permissions. It is a step towards removing the requirement for rodsadmin privileges to deploy Metalnx.
Additionally, the pluggable search feature will now play well with the upcoming new release of the iRODS Indexing Plugin (they will now use/expect the same indexing schema).
You can read more about the upcoming plans in the roadmap
cyverse/go-irodsclient (0.5.3): Go iRODS Client released
0.5.3 of the CyVerse Go iRODSClient has been released.
RENCI Intern Violet finished this month.
You can see their work highlighted in the TriRods talk “Delay Server Availability and Scalability” recorded on YouTube.
Prometheus Exporter
As part of the Lexis Project, Ruben Garcia has developed an plugin for the Prometheus monitoring system to show if iRODS is running.
Main Repository Activity
Open Issues
use of RUNPATH on Ub.16 may prevent plugin components with externals lib dependencies from loading
The libirods_rule_engine_plugin-elasticsearch.so component could not be loaded due to dependencies on libjsoncpp.so.19 (located in /opt/irods-externals/elasticlient0.1.0-1/lib, installed on the system at the time) that were not loading in Ubuntu 16 probably due to ELF not working well with the RUNPATH field of .so objects on that Ubuntu release. All tests failed due to inability to communicate with elasticsearch (the purpose of libirods_rule_engine_plugin-elasticsearch.so when successfully loaded by an iRODS agent process).
Document/Investigate condInput keyword usage in hierarchy resolution
An issue exploring “how different keywords provided to the condInput for a dataObjInp_t-like structure affects various operations via the API endpoints in iRODS 4.2.10+”.
irods recompile and delete directly from filesystem
The issue reads to me like a support request, really, but if you end up compiling iRODS yourself rather than installing from a package, maybe take a look?
Add configuration options to run_tests.py for federation
It is extending the CI tools & test harness. Federation has several edge cases a direct to zone connection doesn’t, so it’s always good to see tests around this area.
iquest fails after (TCP timeout + 60 sec + querytime)
If the network connection drops to your virtual instance (say when it’s been moved between hypervisors), connections in progress at the time are dropped. iRODS does not at this time reconnect to its database in this instance. Unfortunately, due to TCP timeout sessions on both the client (iRODS) and the server (the database), assuming they are on different machines, it can take some time for the session to notice and drop. The TCP timeout sysctl values can tune this - the issue has more details and a series of updates as the matter was investigated.
Blank Origin and Label fields in packages.irods.org Release/InRelease files
Packaging is always hard to do right.
Multiple sources of truth in Provider configuration files
An iRODS server has a source of truth as to which server it connects to - its environment file. If that environment file points to a different server, then that server’s server_config.json` file determines the settings (such as Zone Keys). The configuration makes sense in retrospect but made for tricky debugging at the time.
where to find salt for passwords stored in the database for a new provider
The setup script now allows you to provide a non-default salt for the passwords, which might catch you out if you specify a different one if you are setting up a replacement or additional provider.
Commands for non-admin user to determine users/groups
Should you end-users be able to determine what other users are on the system? Is that a valuable feature for collaboration or a security issue? If you have an opinion, please weigh in on this issue.
iRods RPM breaks common expectations for packages
Packaging is hard, especially for multiple OS’s.
Support SNI when connecting to remote irods server
Workarounds for lack of #5669
This must be against master - it’s not tagged with a proposed release version yet.
Until #5669 is implemented, the irods CMake import targets will not properly pass down all their dependencies. Targets building against them must themselves track these transient dependencies in the meantime.
CMake consistency sweep for 4.2.11
Python3
The critical bit here I think, is Terrell’s reply;
Yes, we are targeting ‘native/packaged’ Python3 support for 4.3.0 (working to get it ready/out ASAP).
This clean break will allow us to support a number of OS distributions that now ship with Python3 by default. We’ll update the dependencies in the master branch.
Add facility for manually refreshing server_config in running irodsServer
There are a few settings that the AgentFactory itself (or perhaps the delay server) read earlier and it’s ‘too late’.
This SIGHUP response/reload would allow the main server to be refreshed without a restart (which today requires killing/bouncing the server or gracefully waiting for all open connections to close on their own).
An additional bonus of this is that containers running iRODS could use the main server as their ENTRYPOINT, which makes docker-compose projects much cleaner to build and maintain.
Add cpu / ram / network usage reporting to main server
Feature request
If you have an in-house monitoring system you would like to see supported, this issue would be the place to put the details in, I suggest. I recommended Prometheus due to its widespread Kubernetes adoption, and relatively modern codebase (unlike the sysadmins stalwart, Nagios).
Design and store latency map
Feature request
Consider having every server within a zone hold a latency map for all the other connected iRODS servers.
Add settings refresher to main server
See 5821, above…
Add periodic lock checker to main server
Feature request
Checks for locked data objects and checks to see if the agent holding the lock is still there.
Related to 4977
Recursive irepl silences failures
Add support for additional read-only database instances
iput across federation does not report CAT_NO_ACCESS_PERMISSIONs
Recently hit this in baton also.
document how to use ‘between’, ‘AND’ ‘>’ and ‘<’ in iquest
setup_irods.py should support high availability (HA) (multiple providers)
An update from Terrell after I ran into this recently;
This is the currently expected behavior - out of an abundance of safety/caution. If setup_irods.py sees an existing Zone, it stops.
That said, if the server_config.json is populated correctly, you should be able to start the server manually (./irodsctl start).
We are planning to add HA-aware support to setup_irods.py in 4.3.0. This will mean we need to ask at least one additional question (which will affect any automated .input or JSON inputs you may be building/using).
containerize irods
An old issue around how one starts up and configures an iRODs sever - it looks like one of the developers has been assigned it. Coincidentally, the developers who seems to be doing the most recent work on containers. Interesting!
create irodsMonitor binary
The update to this issue is how I found out about the Prometheus exporter mention in the ‘news’ section.
Path to “trash” should be parameterized
It looks like this is being taken off the active development plan. It’s an old issue, but if it’s important to you, do let the developers know.
move/copy of file between federated zones should copy metadata values
Closed Issues
closed on - 2021-09-08 00:22:04 PEPs are not being triggered on the master branch
closed on - 2021-08-24 16:52:53 codacy sweep 2021
closed on - 2021-08-19 12:30:03 upgrade from iRODS 4.1.11 to 4.2.8 and then 4.2.10 leaves you with two versions of irods-externals-nanodbc2
closed on - 2021-08-25 00:51:29 Add support for optional server configuration service endpoint
closed on - 2021-08-23 22:32:55 Stopping iRODS on the host will kill iRODS servers running in a container
closed on - 2021-08-30 13:01:42 Add facility for asynchronous tasks in the main server
closed on - 2021-09-13 11:56:37 Some help with Custom Rules
Python iRODS Client Activity
Open Issues
Rule execution should allow specification of a rule-engine instance
irule has -r INST_NAME, whereas currently in PRC instances cannot yet target a rule engine instance by name.
Put operation ignores kw.FORCE_FLAG_KW flag
Put operation over an existing object raises a KeyError (-406000 locked data object)
Not even a resource hierarchy is necessary; actually, parallel_put is not selecting the correct replica on one or more threads.
DataObjectManager.open() ignores DEST_RESC_HIER_KW
This ticket was updated to reflect the investigation in the above issue, I think.
Closed Issues
closed on - 2021-09-09 14:49:23 client API endpoints don’t convey back an RError stack
It would be convenient if the PRC library could facilitate the conveyance (to userapplication), or at least logging, of messages coming back from the iRODS server as part of the response to an API call ( say a checksum-with-verify, or a rule exec )
And then;
This has been done for chksum and can easily be done for rule execution, too. If anything resurfaces anew within that effort, will re-open this issue.
…
let’s make a new issue if we need rule-execution-specific stuff - can always link to this issue in that description.
closed on - 2021-09-12 01:19:07 iRODSDataObject method **options docstrings
Some discussion around “what values are available or legal for use for some methods of iRODSDataObject”.
closed on - 2021-09-12 01:20:17 SSL certificate verify fails even if disabled in environment file
it appears that this library is not currently paying any attention to the “irods_ssl_verify_server” value in the environment file during ssl_startup()
Merged into master, I don’t think there has been a newly released version with this in yet.
closed on - 2021-09-09 15:07:23 error importing iRODSSession
It appears prettytable was bumped to 2.0.0 in November, and this broke something. 1.0.1 still appears to work
Dan then included a recommended VirtualEnv setup to cope better with the vagaries of supporting Python2;
$ pip3 --version # should yield something 8- or 9-ish unless you've upgraded; if not, a "rm -fr ~/.local/lib/python3.5" should revert $ pip3 install --upgrade --user pip==20.3.4 # --> known sane version of pip $ python3 -m pip install virtualenv $ python3 -m virtualenv /tmp/py3env ; source /tmp/py3env/bin/activate (py3env) $ cd ~/python-irodsclient(v0.8.5) ; pip install .
It’s also worth noting that;
We are planning to drop support for Python2 with iRODS 4.3.0 and python-irodsclient 1.0.0.
iRODS 4.3.0 will be as soon after 4.2.9 is safely possible, hoping by June 2021 (iRODS User Group Meeting). iRODS python-irodsclient 1.0.0 will be as soon as we have parallel transfer landed and some community members have had a chance to break it for a couple months.
and finally, the issue was closed with;
Also worth noting, substituting <21.0 for ==20.3.4 in the pip version may be more future-proof - reason being version 21 is the established cutoff point for Python 2 retro-compatibility.
closed on - 2021-05-14 20:21:12 Making a Jenkins dot for the Python client
closed on - 2021-09-09 01:59:22 dynamic metadata PEP not invoked adding AVU to collection
originally this issue occurred while testing the indexing plugin , but is now not reproducible.
closed on - 2021-09-09 17:53:21 Integrate test harness, allow test suite to create an .irodsA for PAM auth
NFSRods Activity
Open Issues
Closed Issues
icommands Activity
Open Issues
Closed Issues
externals Activity
Open Issues
Closed Issues
YODA Activity
Open Issues
Closed Issues
closed on - 2021-09-16 13:37:51 [QUESTION] Problem with installation
I’m not going to try and summarise this (partly because I suspect I will be revisiting in the future when I finish deploying my POC!), but there are bits here around deploying it from 1.7, using the allinone method.
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
Three Yaks were shaved in the making of this newsletter.