Brainstorming Scientific Experiment Designs

[Juan E. Sandoval]

Hello,
I wanted to start a brainstorm thread with you all, especially those of you with some academic training related to statistical analysis and experimental methodologies. Let me know what you think of these ideas, or if you have anything to add too.
Limitations & Considerations - My proposition is that CT can work to design and fund its own first experimental studies, with a limited, but properly controlled sample group. Not under any university or institution, but as a self-funded independent research lab. Do let me know if any of this is impractical, but what I'd like to propose is:

Setup

1. A ~100 sample set of randomized adults.
The idea would be to get (as close to randomly as possible) ~100 adults from the general population, between ages 18-65, to participate in the study. They would come in for a (recorded) interview + a psychological questionnaire, and leave with some compensation. I realize compensation, as a strategy for getting participants, is itself a problem of selection, but it seems the lesser of all evils to me.
Now, from what I can tell, getting these 100 random stranger interviews seems like the most difficult part, since it would require someone to do the scouting, to do the recordings, and collect the surveys. But once we have the raw video data, everything else is fairly straightforward.
2. A new version of the vultology code, designed to be wholly dichotomous in its signals, as well as exact in its measurement criteria, would be used to analyze the 100 videos, second-by-second, for 10 minutes each.
3. A psychological survey would be designed, and given to these 100 participants after their video recording, and be composed of some 120 questions that measure the core elements of cognition we track (10 questions per function, 10 questions per energetic quadrant = 80 + 40), and some 30 that measure personal details such as career, lifestyle, political orientation, etc. In total it'd be a ~150 question survey.
Each of these three elements above will take a lot of work to prepare/complete, but I think that even just the exercise of trying for this experiment will pressure mental energy into developing a greater methodology.

Data Analysis

After the data is collected, we can do anything with it. We'd finally have a clean/pure sampling of 100 people, with little or no bias in selection, and see what we actually find. We've been needing this since forever. And since we'd own the data (that'd be part of the participant deal), we can test it from as many dimensions as we'd like.
However, chiefly, this experiment would be aimed to measure the efficacy of the vultology code (3.0) and psychological surveys. Through it we can examine, in a preliminary way:

• If, for the most part, the function axes are a bimodal distribution under controlled settings. (or not)
• How much signal mixing there really is.
• If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency.
• If there is any correlation between the psychological survey results and the vultology code results.

This would simultaneously act as an experiment to know if CT's truly onto something non-random, but also to test where there may be weaknesses/strengths in the current rendition.
But the purpose of this experiment would not be to justify the fundamentals of the model, in any rigorous sense yet. It would be to test how it functions, as a whole, under a controlled setting. Naturally, the assumptions made in the formation of the vultology code would need to be justified in other studies (such as studies that can generate an atheoretical extraction, from lots of data, of the most common body-movement-clusters in people). But this would only be necessary if there's something genuine that's discovered by the model as it exists.

Lastly - How to fund it?

Of course, we all know that research studies can range in the tens to hundreds of thousands of dollars. And it's a bit of a pipe-dream to wish to do something like this without a grant. But I personally think a self-funded study like this can be done for a few grand, by working within limitations and not expecting to be super formal about it. It would not be aiming at scientific publication yet, but would be a path-finder study, to learn what does and doesn't exist. I think applying for grants may be something to think about later, if it turns out there's something legitimate here which can be tested at much larger scales. For those who work with grant applications, what are your thoughts?
Assuming it has to be funded out-of-pocket, I'll need to hire a few people for this. I'm currently not in an English-speaking country, so I couldn't do the interview recordings, or survey administering, myself even if I wanted to. Someone will need to be hired to perform the recording of the 100 strangers, and all that comes with that.
What I'm currently thinking is that, by the latter part of the year, after generating a wider online presence (through the new video series), I'll make a video discussing the finalized idea, along with a crowdfunding campaign. The money would also go to commissioning statisticians/etc.
And the number of samples we end up doing would depend on how well the campaign does. If it does better than expected, we can get over 100+ people recorded, and shoot for 150+. If it does poorly, we can just do 50 or 30. That way the study is flexible and won't bomb either way.

Thoughts?

What are your thoughts on this?
Would you change any parts of the experiment above?
Thanks!

[Supah Protist]

I think a big thing to keep in mind is quantification. I’m not sure that measuring people’s opinions on which signals someone is expressing qualifies as objective data. Setting up a protocol that only records the the quantitative correlates of the signals may be more reliable. For example, actually measuring the number of discrete shoulder shrug movements and comparing this to the number of discrete head nods as well as the times at which they occur. This would require forming perhaps quantitative but definitely discrete definitions of shoulder shrugs and head nods in this case. I don’t think setting up a dichotomous signal library is necessarily the best idea because it would rely on qualitative descriptions and interpretations as opposed to quantitative data that could speak for itself.

[Juan E. Sandoval]

Heya Supah!

For example, actually measuring the number of discrete shoulder shrug movements and comparing this to the number of discrete head nods as well as the times at which they occur.

Yes, have you seen the new multi-timestamp codifier with .CSV file export functionality? https://cognitivetype.com/forums/topic/multi-timestamp-support-for-codifier/
My thinking is quite in agreement with yours, I think. The new codifier allows for individual signals to be tallied, at every second, so u can get a numerical count of how many of each signal was displayed, and work with that data via Excel/etc.

I don’t think setting up a dichotomous signal library is necessarily the best idea because it would rely on qualitative descriptions and interpretations as opposed to quantitative data that could speak for itself.

Well, the dichotomous setup would be necessary to see if what is being tested is a true dichotomy or a bell curve. And I think that's a separate topic to the topic of qualitative-vs-quantitative.
As for quantitative vs qualitative, there was some discussion of that in this post, I wonder if you saw it? https://cognitivetype.com/forums/topic/is-the-vultology-theory-somewhat-self-confirming/#post-19671 - In short, the idea would be that a new version of the code would be made (3.0) specifically designed to be fully quantitative.  Every signal would be registered as a motion vector, and be non-contingent on any other signal, so that correlations aren't piggybacked onto each other artificially.
This would allow for us to properly examine whether or not the signals currently grouped under (lets say) Fi are all statistically connected to each other, when structural contingency is removed by consolidating them into non-contingent signals. And also whether that group (if it exists as a discrete set) correlates to Te (as a discrete set). And whether Fi+Te is negatively correlated to Ti/Fe as a discrete set.
I wonder what you think?

[Supah Protist]

I think what I meant by “dichotomous signal” is referencing the interpretative nature of the reading that would be done. The dichotomy can be used to interpret the data, but I don’t think it should be used in gathering the data. Setting up a dichotomous signal library makes people choose between two options, which would mean that they may bias the aggregate results. If people are looking for Te 1 vs. Fe 1 they may potentially be biased towards seeing Te in general or Fe in general if they already know what they are looking for.
A key issue is that there aren’t quantitative definitions of the signals at present. If you were to use the current qualitative signal definitions, it would be hard to saying anything of value about the patterns in the “data” because the “data” would actually be subjective. Many languages have a word for red, people can identify red things reliably. However, red is not a discrete property of any object. It is an arbitrary subregion of a spectrum. What I’m saying is that there’s no science of red or color itself because they are subjective concepts as opposed to an objective ones. There’s no discrete transition that occurs between red and orange light so you can’t study the difference objectively.
Saying “we all consistently identify this apple as red” doesn’t mean much if the terms haven’t been defined quantitatively. What do you mean by apple? What do you mean by red? Better statements may be “this apple weighs x ounces” or “this apple contains carbohydrates”. This is because these statements can be empirically validated through quantitative methods. If you use subjective definitions for the signals then the relations between the signals won’t really mean anything.   I guess the main point is that what people think something is doesn’t count as objective scientific data since it’s not a quantitative measurement. I hope I’m being clear enough.
Also, the survey will be important since none of these vultological correlations matter if they don’t correlate with behavioral and psychological variables.

[Juan E. Sandoval]

Oh, I don't think the goal of this idea is to be that reductive.
Granted, I don't think you're wrong, but the level of specificity your proposal would require is astronomical. For example, a whole study would have to be dedicated to identifying when and how something counts as a head nod or shoulder shrug [or anything else]. How much lift of the shoulder counts as a positive mark, or how much accentuation of the chin counts as a nod? With what velocity, and from what angle? Is it measured in inches? Inches per second? Does one inch not cross the threshold, but three does? Or is it proportional to the person's body size? Etc.
(And then there's the infinite variability in human mannerisms overlapping. For example, what if they're leaning their arm on the chair arm-wrest, which naturally lifts their shoulder 2 inches. Do we count this as a shoulder shrug, or is it only in relation to a certain baseline from which it rises and falls quickly?)
By this sort of logic, cataloguing human movement into discrete signals, objectively, either cannot be done, or it would take training a neural network to identify clustering (as I mentioned) -- so long as we accept the results of a neural network itself. But then that's contingent on the training data, which can vary. So we'd have to decide at some point how much training data is good-enough, what data is good enough, and the list goes on.
I've been keeping up with the progress on machine learning, but I haven't seen anything that has catalogued natural human movement accurately, yet. And this is with well funded labs. So there's no chance for CT to do something like this anytime soon. Therefore, we have to make concessions.

Concessions

What I'm proposing is that we allow for some general assumptions, but have those assumptions be informed by quantitative descriptions.
So for example, lets say that we define a Je-5 Projecting Hands gesture as a forward protrusion of the arm where the elbow is in front of the torso area, the wrist is in front of the elbow, and the elbow angle is no less than 90 degrees wide. We have this as a verbal description but we don't actually capture the sample from multiple camera angles and go measure the degrees in the elbow angle at each frame. We use human eyes to approximate these values.
I find this level of granularity in describing signals, as motion vectors, is acceptable for our purposes at the moment, no? Once again, the aim wouldn't be to get scientifically published (precisely for reasons that you mentioned), but to perform a pilot study that can make a few concessions on the nitty gritty, while getting an overall sense of whether something is there or not.
What do you think of something like this, or do you think that any concessions of this degree would make the data useless?

[Supah Protist]

I think I may need to contextualize this discussion.
Experiments test falsifiable hypotheses.
What falsifiable hypothesis do you want to test?
If there are several, feel free to list them.

[Juan E. Sandoval]

Well, chiefly the ones listed above as:

• If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency (and moving to code 3.0).
• If, for the most part, the function axes are a bimodal distribution, under controlled settings, or not.
• If so/not, how much signal mixing there is.
• If there is any correlation between the psychological survey results and the vultology code results.

But other things could also be evaluated.

[fayest42]

I wonder if an artificial neural network could be helpful? I don't know much about them, but perhaps an ANN could learn to type people if we just fed it a bunch of videos of people that we've typed. Then we could give it different videos of other people we've typed to see if it categorizes them the same way we do. If the ANN consistently assigned people the same type that we do, then we would know that there are objective signals. Plus then we'd have a have a way of typing people that didn't rely on humans, which would be quite useful in a variety of different ways. I'm not sure how ambitious something like that is, but it doesn't seem totally outside the realm of possibility.
Edit: Oh, I didn't see some of the posts above before I posted this. Looks like you've already had this thought 🙂

[Juan E. Sandoval]

@fayest42 It's a great thought though!
And yes, precisely. The practical applications would be enormous. It would have to be trained indeed. I was talking to Staas about how, if we refine the codex, then generate some 100 CSV's of 100 videos, as the starting data, we might get a level of predictability. But yea, I don't know how involved this would be either.

Spoiler

I came across this today:

It's used for sign language recognition, but the programming structure might be similar enough to be repurposed. Not sure. The guy put the code up in GitHub.

[collapse]

[fayest42]

@Auburn That sign language translator looks really neat! I don't know enough about programming to know whether the code might be useful to us, but perhaps.

if we refine the codex, then generate some 100 CSV’s of 100 videos, as the starting data, we might get a level of predictability.

Forgive me if I'm wrong about how ANN's work, but isn't it true that you wouldn't need to actually teach them specific signals through the csv's? Wouldn't you be able to just give it a bunch of videos and tell it what type each person is and then the ANN would figure out for itself how to recognize different types?

[Juan E. Sandoval]

Oh I don't know too much about it either. I only just started diving into them. I think there are various methods and one is called 'supervised' learning, where an initial training set is utilized. Neural networks can be trained at different levels of supervision, I think. Seems different approaches work for different use cases.
If the only feedback the neural network got was whether it got the type right or wrong, that might not be the best feedback for it to learn. Or it could be, but generally the less supervision there is, the greater the sample size has to be. I'm not sure if we'd need thousands or tens of thousands of videos to train it. But each video is thousands of still images, so I have no idea how intensive that would be.
Another thing to consider is... most work on AI networks and visual recognition that I've seen deal with properly labeling static photos, which is hard enough just on its own, without stacking anything else on top of it. So for a neural network to even recognize that there's a person in the video, is its own challenge. So is anything nuanced like if they're wearing a certain shirt, and at certain frames the shirt shape or color makes it ambiguous to tell it apart from the arms/etc.
We'd run into a lot of other basic visual recognition problems before we can even test out the CT hypothesis. So I think a more targeted optimization would be needed for this, to help simplify the matter. Something like full-body skeleton tracking:

Spoiler

[collapse]

If something like a Kinect (which needs no physical markers) can be used to track the body, then the raw data for the joints can be extracted. Once the raw data is there, we can run just those values through the neural network like you described, giving it minimal supervision. I just think that having it solve the computer vision problem at the same time as everything else is too much.
But then, if Kinect is not accurate enough, this might mean that our samples would need to wear body trackers!

Spoiler

[collapse]

Haha, that'd be a fun surprise for our 100 strangers, eh? "Here, put this on!"

[fayest42]

Neural networks can be trained at different levels of supervision, I think.

Ah, that makes sense. In that case, some amount of supervision could be useful. But I'd also be curious to see if it could figure out signals that we haven't noticed.
Re: kinect, yes, that makes a lot of sense. I imagine it would be much faster and would require fewer samples for it to learn just from raw movement data rather than from videos. Heh, body trackers could be fun 🙂

[xaiouhs]

Okay, so I wrote a long post about this as I work in deep learning. Yet, I'm editing this after thinking about the subject for a few minutes. CT does not have the data for deep learning. You need probably 100 examples of each vultological signal in a short clip (1-3 seconds each?). With enough work, you could have the algorithm trace the right muscles and parts of the body. But without a whole lot of samples of each signal, you're not going to be able to get an algorithm to learn the appropriate quantitative measures. It's the best method to measure signals using an algorithm, but after some more consideration, I really don't think the data problem is manageable with the current team.
Your best bet is to have several people independently quantify each signal, then statistically measure how much their evaluations correlate. For instance, two people select Fi-2 within 3 seconds of the same time. That's a sign you're actually measuring the same thing.
I edited out most of the unimportant stuff from the original post, as it was mostly written humorously or as an offer to work on a project that I'm not sure is feasible. The remaining information is below.
---
A completely novel theory of human cognition is not going to fly for scientific papers. You're going to need to pick out something simple, such as the clustering of certain facial features. Then in another paper, you apply a cognitive assessment/test showing that there's a correlation between these facial clusters and psychological phenomena. I'm sure you have already considered this.
For the eventual psychological assessment, whenever it is utilized and/or published, you want to closely follow the systems used by other personality assessments, such as FFM (without using FFM itself of course). While the tests obviously have to be tailored to CT, the more you can use already tested material, the better off you are. This is of course, if your goal is for CT to be taken seriously in scientific literature. Obviously, you can just develop your own questions based on your own thoughts on accuracy, and quite possibly get better results. You just might not get published.
Funding for participants is generally acceptable, though there are standards for how much should be compensated. There are also methods for how to select a more or less random population, though most studies do primarily utilize college students or other limited groups.
Let me know if you want to discuss this more over voice chat, or if anyone has any questions in particular. I'm extremely familiar with deep learning and psychology.

[Juan E. Sandoval]

hey Xirailuyo! Thanks for dropping by. 🙂
Getting 100 clips per signal is a hell of a grind, but doable if we wanted to go that route. We kinda need that anyway.

A completely novel theory of human cognition is not going to fly for scientific papers.

Naturally.
I've been aware of this from the very start, which is part of why CT took this independent research path-- because the development of the model requires associative and holistic (non-reductive) thinking for it to be conceptualized (and refined) as a whole during the early stages. Much like a neural network learning how to 'see', CT started off by a group of us having a 'fuzzy' view of the entirety of human motions/mannerisms, in order to properly orient itself across the aggregated thresholds dividing all its phenomenon. We thus acted a bit like this neural network and fed ourselves thousands of videos to extract patterns. Things like Je, Ji, Pe, Pi manifest as clusters at this level of holistic/panoramic resolution, as strong natural grupings of signals.
Then, verifying or falsifying these conclusions would each take a whole independent study. But the idea is that this would be repeatable, and that other neural networks that examine human motions would also cluster human energetics first around Je/Ji/Pe/Pi, or something very close to it.
I think this can be tested, through deep learning + body tackers, if I'm not mistaken Xirailuyo? Could you think of an experiment where a neural network could be fed body-tracked data of people sitting on a chair being interviewed, and see if the motion vectors fall into discrete categories?
What do you think of something like this, as one possible test of the energetics?

[Sander]

[quote=@auburn]

• If, for the most part, the function axes are a bimodal distribution under controlled settings. (or not)
• How much signal mixing there really is.
• If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency.
• If there is any correlation between the psychological survey results and the vultology code results.

[/quote]
You already know the statistical significance of your first and last hypothesis. The recent quizzes already showed statistical significance, and the next quizzes can verify further improvements. And the cybernetic Big Five statistics already show the psychological relevance of clusters that were intended to map on function-attitudes.
Moreover, you expect energetics to be primary; so, then bimodality shows up more when functions are developed. However, most test subjects are expected to have only one out of four functions developed; so, the average separation of function-axis isn't a great measure for differentiating CT personalities.
Hence, I suggest testing the energetics first; apart from greatly reducing complexity, that also makes good PR material for crowdfunding a follow-up experiment.

[LadyNerdsky]

Cool, I'm excited by the idea of formally getting more data from the project.
SETUP
Point 1.

• What populations? Doing Caucasian, vs PoC vs Asian in equal proportions? Treated as separate populations? Or solely analyse Caucasian sample to begin with? Is the Vultology code robust enough to confidently handle challenges in signal detection on morphologically different faces e.g. analysing Caucasian vs Asian eyes with ocular tension, slanting edges and so forth.
  • In addition to video, what about also getting static well-lit photos of each participant at a standardised front-on angle, one with a neutral face (plus one smiling, one frowning for example). Might benefit future analysis avenues within the same group: creating facial morphs, measurements of ratios of facial features across types while making expressions, attempting to numerically quantify some signals using proportions. I know this is very granular, but takes no time to take a snapshot while you've got participants right there.
  • Will you have a single, consistent camera set-up, or rely on participant submissions? The advantage of having people attend one location for filming means the angles remain constant. Helps avoid issues with peoples eyes looking more Gamma/Delta depending on whether filmed slightly from above or below rather that straight-on. Using a DSLR will get excellent screen grabs for static demonstrations of signals.

Point 2.

• My only concern is that it’s starting from the premise/assumption that Vultology is measuring cognition, and that face and body expressions are inextricably linked to specific cognitive functions. Is there anything that can lend evidence to the theory that metabolism is linked to vultology before vultology gives rise to behaviourism? The idea of coding videos and referencing with personality tests helps research the link between vultology and behaviourism, but I can’t help but think we’re still missing that first key step. Possibly outside the scope of an independent, self-funded project (going back to EEG/brain stuff).

Point 3

• Will the 10 questions per function be once you're "sure" those questions parse out the differences between functions during the preparation and refinement stage, or is this a broad exploratory survey collecting traits for correlations and convergence on type. If you cast a wide net, you can use Principle Component Analysis / Confirmatory Factor Analysis to see how answers cluster together (what behaviours might be N vs S, J vs P, Fi vs Ti) https://en.wikipedia.org/wiki/Confirmatory_factor_analysis Actually executing one of these is beyond my ability these days, it’s been 15 years.

“In statistics, confirmatory factor analysis (CFA) is a special form of factor analysis, most commonly used in social research.[1] It is used to test whether measures of a construct are consistent with a researcher's understanding of the nature of that construct (or factor). As such, the objective of confirmatory factor analysis is to test whether the data fit a hypothesized measurement model.”

• Would you use yes/no, True/False or a Likert scale for ratings? Including qualitative open-ended boxes? For example, you’ll get a different picture on the career question if you ask people a single forced-choice tick-a-box of their current career area (e.g. Arts, Science/Engineering, Finance) versus getting them to write all their major career interests and changes in chronology, especially in the older age groups. You’ll catch us pesky Ne-types flitting around from broad interest-to-interest.
  • Road-testing the Psychological Survey on the already typed group of us to check the answers correspond to our types in a patterned way. No point rolling it out to paid participants if there’s too much random variance on the psychology survey amongst different types here in the community.

DATA ANALYSIS
Oh good, you’re thinking about if the functions are indeed bimodally distributed and the prospect of signal mixing. And okay, not really having the goal of testing the fundamentals just yet. Like I said on the Discord server, what if there's a mysterious extra axis that we just haven't discovered yet? Another bimodal functional axis that us confusing Fi-Ti hybrid types belong in, Xi/Ye.
But also to reiterate the WSS Socionics dude's point, why throw money, time and effort into collecting 100 video samples and doing the psychology questions if the original Metabolism/Vultology premise isn’t justified? Is the hope that strong correlations at this phase implicitly help with evidence of the former?
Just read your replies to SupahProtist and Fayest. I see your point, but also want to keep adding a reminder.
GENERAL
I can also see what SupahProtist is saying regarding signals, I think. For example, I have a hell of a time telling the difference between an Ni-scowl and a Si-scowl. I’m more likely to be biased into “seeing” Si if I’ve noted a lot of Ne/Si signals so far in the video and no sign of Ni/Se but if you assembled a test of short repeating gifs of Si and Ni scowls randomised and I had to guess which one, I might be at 50% correct against a rater who’s watched the whole video and has context or bias cues.
Same if you got gifs of confusing signals together like the ones referencing slanted edges or ocular tension and got people to rate just the excerpts, are they above chance or are they only reliable signals in the context of a whole video giving out the clues and cues (e.g. Seeing several instances of brush-strokes eyes, therefore it’s likely that their ocular tension is also relaxed under-eye rather than taut, but if you showed the same person relaxed vs taut gifs in isolation would they have the same certainty, or be at 50% chance in identifying which is which).

[xaiouhs]

Here's what you're basically looking at:

See those dots? They're generated by an algorithm automatically, and they're doing just about the same thing as a full body motion capture suit. Sure they're not as accurate, but they're pretty damn close. You can also choose exactly where you want these points to trace, though that is usually done by hand (though any video data of a person will suffice for that. It can also be crowdsourced or even done by community members). If you want to do Deep Learning, your problem isn't generating these points, it's creating categories the algorithm knows. Regardless of whether you use a suit, existing technology, or your own video algorithm, you still need lots of examples of each signal so the network knows what it's looking for. Furthermore, that red box focuses the algorithm on a region (such as the head), and boxes can be created for the arms and shoulders as well. Again, this is all automatically generated as objects come into view. If you tried to go without the categories completely (unsupervised learning), you'd likely pick up on larger categories like smiling or movement. You're much better off training on certain categories from the start, as annoying as it is. Even though you came up with vultology by watching people without pre-existing categories, you started from the basis that you were looking for something, and that you could ignore certain things. An algorithm has no idea about these nuances, or really any nuance.
Clustering around Je/Ji/Pe/Pi is definitely a good idea to get started, along with general J and P descriptors. It would lower the data cost of entry.

[Author]

Posts