Is there a rule governing generation of RTX task ids OS_TID? It would be interesting to know:
a. if they are incremented sequentially, and b. if there is a max value for OS_TID
I am writing a lib which will cache some data in a buffer. That lib can be called from multiple tasks at the same time, so I want to keep separate buffers for each task. If OS_TID is deterministic then it will be a good differentiator for the per-task buffers.
Any other suggestions are welcome too.
Thanks
Anyone?
I can't see the need for them to publish their method. Certainly never seen any mention of one.
Not publishing such a thing means that they are free to change it with the minimum of fuss in the future.
It would be way safer to assume it is unpredictable in any code you write.
Thanks Salty.
In that case is there an efficient way to store task specific data in a function that is called by separate tasks?
A hash can address on an ID even if the ID may span the full numeric range of the data type where it becomes too expensive to use a vector.
If a task ID is reused, you can handle that by having each task first call a method register_task(tid) - so if the task identifier was reused then you can clear the hash of the cached data from the previous thread using that same ID.
If you know that a task is about to end, you can have it explicitly release any cached information by calling a function unregister_task(tid).
It's possible to implement a hash container without use of dynamic memory since you already know how many threads that are likely to exists at the same time. This allows you to preallocate the data structures needed.
In my case a different task with same id is not a problem, so I can get away without register/unregister API.
Since I know the maximum number of tasks that will ever exist, I guess I can implement a simple modular hashing with linear probe. Since the task ids generally tend to be linearly increasing this should be pretty efficient in both execution and memory. Even if they change sequence number of OS_TIDs it should still be pretty good. Worst case scenario will be if they OS_TIDs increase by
MAX_TASKS
where
is maximum number of tasks that will ever be created, because modular hash will really be
OS_TID % MAX_TASKS
. That worst case is too arbitrary to be likely.
Any thoughts?
a. if they are incremented sequentially Not exactly. There are enough TCB entries for every task. The TID is allocated as the "next" available entry. If you never delete tasks, they will be incremental. As already stated, this is just the way it is now, not guareentee it will stay this way.
b. if there is a max value for OS_TID At the moment yes, it will not be larger than the number of tasks allowed. May change in the future.
I was refraining from answering because I was hoping you might find a better way to do this. I think you have probably spent too much time on this.
The RTX uses a concept to allow the C runtime library to be thread safe. Each task has a block of RAM associated with it for use by the C runtime library based on the task running. You could add to this block for your own purposes.
//This is in RTX_CM_LIB.h, which is is included in RTX_CM_LIB.c so // even though RTX_CM_LIB.h resides in with the OS_CODE, it is not part of the // built OS library, it is part of your code. /* A memory space for arm standard library. */ static uint32_t std_libspace[OS_TASK_CNT][96/4]; static OS_MUT std_libmutex[OS_MUTEXCNT]; static uint32_t nr_mutex; extern void *__libspace_start;
One way would be to add your own block of memory here, and provide a function to return a pointer to the memory based on the taskID like they did for the C runtime library. Notice they also added mutexes for use in the few cases that separate memory does not make it fully thread-safe.
// Also in RTX_CM_Lib.n void *__user_perthread_libspace (void) { /* Provide a separate libspace for each task. */ uint32_t idx; idx = (os_running != 0U) ? runtask_id () : 0U; if (idx == 0U) { /* RTX not running yet. */ return (&__libspace_start); } return ((void *)&std_libspace[idx-1]); }
This answers your 2 questions and shows a away to do as you describe. It has been noted that they the TID implementation may change in the future, but Keil has already taken advantage of how it is implemented in implementing this C library RAM for each task.
Now some comments. The C run-time library was designed with the thought of being thread safe. Some functions were very clearly not even close to thread safe. They were expected to have exclusive access to data and run to completion before something else could call the functions. This is a reason why someone might choose to had separate RAM per thread as well as mutexes.
Writing you own code from the start would make me think that you could just make it fully thread safe and re-entrant without using this kind of "trickery". You see that having a library (such as the C run-time library) not being fully self contained causes the user to make code that lives outside the library just so that they will function.
This might be a case to consider a c++ object. They can have member functions (library function calls), member data unique to each instance of the class as well as both static data and member functions that can be used as appropriate.
This is an interesting reply. First, given that ARM's C runtime relies on OS_TIDs being sequentially increasing, it is a shame that there is no explicit commitment from RTX. Even Unix systems which are much more complex, make process ids sequential.
Regarding your suggested approach, I find it cumbersome and intriguing at the same time. Cumbersome because of the semantics of such an API. If I understood correctly, first you need define per-task buffer and then have the calling task pass in a pointer to that buffer to the function that actually wants to keep track of per-task data. E.g. in a logging API which likes to buffer some data before flushing to file, the calling task will have to pass in a pointer to its buffer to log() function. That creates unwanted dependencies and tight coupling between caller (task) and callee (log() function) - a fertile ground for breeding spaghetti code. Moreover, buffering is responsibility of callee. Why should the caller worry about it?
Nevertheless, it is intriguing because it touches parts of RTX runtime. I would like to dig into the runtime to understand it better. I couldn't find its source code - can you post a link?
"Even Unix systems which are much more complex, make process ids sequential."
An interesting statement.
First off - you need a huge table to handle all PID of a modern Unix system. And that Unix normally have lookup tables because it needs to perform translations from the "sequential" values.
If you read the Linux man page for getpid(), you'll get an indication how hard that function has to perform the task - just because it caches the value to avoid additional system calls. And it's not trivial for a C library to keep track of the unique memory regions for different instances.
Next thing - your Unix system will not promise you what PID a new process will get. The promise is that it will get a PID which will fit in the pid_t data type. And that will not collide with the PID of any other, currently running, process. A new PID can be higher than the PID of the calling process. Or it can be lower.
Have you, by the way looked at the complexities involved in thread local storage? Lots of seemingly trivial functionality is way more complex when you do look under the hood.
"If you read the Linux man page for getpid(), you'll get an indication how hard that function has to perform the task"
But RTX is already doing it! OS_TID's implementation, as it stands, is exactly what we want. Only thing that's missing is an explicit commitment from RTX that this won't change in future. Unix makes that promise.
"Next thing - your Unix system will not promise you what PID a new process will get."
And I'm not asking for that.
I'm not in any way trying to trivialise the effort involved in building RTX. But holding back on solidifying an aspect of the system - generation of OS_TIDs - makes life harder for users. It feels even worse when the runtime itself relies on it. I gave the Unix example to argue that such a commitment can be made in practice, not on how to implement pids :)
"Have you, by the way looked at the complexities involved in thread local storage?"
The question doesn't ask for TLS. It asks whether generation of OS_TIDs is reliable.
No. Unix does not make any promise about the sequence of numbers in use. The only promise you have is the data type pid_t. And on my machine, that value is a 32-bit integer. So if you keep an array with the PID as index, then you need to be prepared for having 2^32 array entries. If we then assume that you can manage with 16 byte per possible PID, then you would "only" need a 64 GB large table.
The machine also happens to have an entry /proc/sys/kernel/pid_max with the value 32768. But that value is not a commitment to how Linux will work - it's just information how the specific implementation is currently configured. If I switch Linux kernel, I can't assume the value will be the same. I can't even assume that there will even be a pid_max file to read, or that the kernel will have any maximum value lower than what the pid_t data type can store.
The POSIX requirements is basically that a PID is a signed identifier not larger than the long data type.
So in a Unix system, you can't assume that you can get away with the PID as an index into an array because you have no promise that the possible range will be just a subset of what the pid_t data type can store. Not only that - you can't even assume that the PID you read out is the actual PID of the process. You can have translation layers that remaps the PID values to allow the use of sandboxes where a specific process will only be allowed to see a local mapping of a small subset of the full number of processes in use. This allows applications to run in firewalled containers without requiring virtual machines.
Why should RTX have to make a commitment how to actually allocate thread identifiers, when you use Unix as an example - and Unix does not make any such commitment? An additional commitment means limitations in what improvements that may be introduced at a later time. So commitments should only be made when it is really important to lock down some functionality.
If an OS promises you a nice integer data type of a size that works well with the programming language/hardware architecture, then that really is all commitment that is meaningful. Any additional commitment will be counter-productive because it will permanently add additional "must" clauses without giving any gain for the huge majority of the users of the API. If Keil finds a way where they can cut 100ns from a task switch, then it will be a big advantage to a lot of customers. If that 100ns task switch improvement clashes with any additional thread-id commitment, then lots of users will fail to get that 100ns gain because of the 0.1% users who could actually gain anything from additional commitments of how thread identifiers are allocated.
There is a reason why the software industry focuses on black box designs, where the implementation is allowed to change as long as the new implementation continues to fulfill the promised functionality/performance.
Just an addendum. The code to find a free thread ID is almost certainly deterministic. I don't think they are using a pseudorandom number generator when locating a suitable ID value. It's just that Keil has no reason to document how the code finds a non-colliding value within the possible number range. And Keil has no reason to document the possible number range in any other way than to document a data type for storing the values.
So in the end, neither *nix nor RTX relies on the ID:s being sequentially increasing. Just that they can handled by the intended data type, and that all active ID (you can see) are unique.
"Just an addendum. The code to find a free thread ID is almost certainly deterministic."
Yes. But it's not promised to remain same so users can't rely on it. Unix promise to keep way pids are generated same so they can be relied on.
Regarding your suggested approach...</p>
I thought you were suggesting having separate memory block for each task that was going to use the library. I am not suggesting this at all. My suggestion was to just make the library thread safe and re-entrant.
Cumbersome because of the semantics of such an API. If I understood correctly, first you need define per-task buffer and then have the calling task pass in a pointer to that buffer to the function that actually wants to keep track of per-task data. E.g. in a logging API which likes to buffer some data before flushing to file, the calling task will have to pass in a pointer to its buffer to log() function. That creates unwanted dependencies and tight coupling between caller (task) and callee (log() function) - a fertile ground for breeding spaghetti code. Moreover, buffering is responsibility of callee. Why should the caller worry about it?
No, you did not understand at all. The application does not need any knowledge of how this is implemented. (My guess is that you probably had no clue the Keil was doing this for the C runtime library. IF they changed the API for the C run time library my guess is that you would have had a clue about this and thought they were insane for changing the interface. Only the library uses this buffer. When the library needs to use one of these per tasks buffers, it uses the TID to get a pointer to the buffer.
this function above returns the unique buffer associated with the task that called the library. The API for the library does not change.
// There is nothing extra to pass in printf() { TaskUniqueBuffer = __user_perthread_libspace(); // This task the currently running task, which by definition needs to be the currently running task..., then returns a pointer that is unique for that task. The API does not change .... }
That being said, I want to re-iterate that I am not suggesting using this method at all. I thought you were looking for something like this. You had said....
"I am writing a lib which will cache some data in a buffer. That lib can be called from multiple tasks at the same time, so I want to keep separate buffers for each task.
and that is exactly what this does.
My interest was to provide knowledge on how some things worked. You had 2 questions about TID's so I answered them. I was not making any suggestions for how you use that knowledge. I pointed out that Keil used something similar to what you were suggesting (I was really not suggesting this) to make the C runtime library thread safe. Since it only required 1 variable and 1 function and no change to the API. My guess is that I would not be choosing to implement it this way, but that does not mean that in your specific situation this is not appropriate.
You can assign, store and compare thread ID:s if using the specified data type. So any code that stores the returned value in a variable of that type will be fine.
Maybe you could explain exactly what additional promise you think you get from Windows or Linux or BSD or OS X? They don't promise that you can predict what the next value will be. And they do not promise if they will use a subset or the full range the data type can store.
This seems to be what Windows promises: "Numeric identifier used to distinguish one process from another. ProcessIDs are valid from process creation time to process termination. Upon termination, that same numeric identifier can be applied to a new process. This means that you cannot use ProcessID alone to monitor a particular process. For example, an application could have a ProcessID of 7, and then fail. When a new process is started, the new process could be assigned ProcessID 7."
On Win32, the process ID is a uint32.
I haven't checked if it has changed on Win64, but the Win64 systems I have observed seems to limit itself to 16-bit values. But I have no way to prove if this is true, without trying to create 64k+ processes - and MS doesn't seem to give any such promise.
What Keil is doing, is quite similar to how thread local storage is implemented - which was the reason I did mention this concept earlier.
And this is because thread local storage tries to solve the similar problem that Keil wants to solve when they make their CRTL thread-safe.
The other common route to make library functions thread-safe is functions like gmtime_r(), gethostbyname_r() etc where a specific thread-safe variant of standard functions takes a pointer to a thread-local structure. Which obviously doesn't work for malloc() and all other standard C functions where you can't break the API.