Brainstorming Scientific Experiment Designs

Home Page Forums General Psychology Brainstorming Scientific Experiment Designs

  • Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    Hello,

    I wanted to start a brainstorm thread with you all, especially those of you with some academic training related to statistical analysis and experimental methodologies. Let me know what you think of these ideas, or if you have anything to add too.

    Limitations & Considerations – My proposition is that CT can work to design and fund its own first experimental studies, with a limited, but properly controlled sample group. Not under any university or institution, but as a self-funded independent research lab. Do let me know if any of this is impractical, but what I’d like to propose is:

    Setup

    1. A ~100 sample set of randomized adults.
    The idea would be to get (as close to randomly as possible) ~100 adults from the general population, between ages 18-65, to participate in the study. They would come in for a (recorded) interview + a psychological questionnaire, and leave with some compensation. I realize compensation, as a strategy for getting participants, is itself a problem of selection, but it seems the lesser of all evils to me.

    Now, from what I can tell, getting these 100 random stranger interviews seems like the most difficult part, since it would require someone to do the scouting, to do the recordings, and collect the surveys. But once we have the raw video data, everything else is fairly straightforward.

    2. A new version of the vultology code, designed to be wholly dichotomous in its signals, as well as exact in its measurement criteria, would be used to analyze the 100 videos, second-by-second, for 10 minutes each.

    3. A psychological survey would be designed, and given to these 100 participants after their video recording, and be composed of some 120 questions that measure the core elements of cognition we track (10 questions per function, 10 questions per energetic quadrant = 80 + 40), and some 30 that measure personal details such as career, lifestyle, political orientation, etc. In total it’d be a ~150 question survey.

    Each of these three elements above will take a lot of work to prepare/complete, but I think that even just the exercise of trying for this experiment will pressure mental energy into developing a greater methodology.

    Data Analysis

    After the data is collected, we can do anything with it. We’d finally have a clean/pure sampling of 100 people, with little or no bias in selection, and see what we actually find. We’ve been needing this since forever. And since we’d own the data (that’d be part of the participant deal), we can test it from as many dimensions as we’d like.

    However, chiefly, this experiment would be aimed to measure the efficacy of the vultology code (3.0) and psychological surveys. Through it we can examine, in a preliminary way:

    • If, for the most part, the function axes are a bimodal distribution under controlled settings. (or not)
    • How much signal mixing there really is.
    • If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency.
    • If there is any correlation between the psychological survey results and the vultology code results.

    This would simultaneously act as an experiment to know if CT’s truly onto something non-random, but also to test where there may be weaknesses/strengths in the current rendition.

    But the purpose of this experiment would not be to justify the fundamentals of the model, in any rigorous sense yet. It would be to test how it functions, as a whole, under a controlled setting. Naturally, the assumptions made in the formation of the vultology code would need to be justified in other studies (such as studies that can generate an atheoretical extraction, from lots of data, of the most common body-movement-clusters in people). But this would only be necessary if there’s something genuine that’s discovered by the model as it exists.

    Lastly – How to fund it?

    Of course, we all know that research studies can range in the tens to hundreds of thousands of dollars. And it’s a bit of a pipe-dream to wish to do something like this without a grant. But I personally think a self-funded study like this can be done for a few grand, by working within limitations and not expecting to be super formal about it. It would not be aiming at scientific publication yet, but would be a path-finder study, to learn what does and doesn’t exist. I think applying for grants may be something to think about later, if it turns out there’s something legitimate here which can be tested at much larger scales. For those who work with grant applications, what are your thoughts?

    Assuming it has to be funded out-of-pocket, I’ll need to hire a few people for this. I’m currently not in an English-speaking country, so I couldn’t do the interview recordings, or survey administering, myself even if I wanted to. Someone will need to be hired to perform the recording of the 100 strangers, and all that comes with that.

    What I’m currently thinking is that, by the latter part of the year, after generating a wider online presence (through the new video series), I’ll make a video discussing the finalized idea, along with a crowdfunding campaign. The money would also go to commissioning statisticians/etc.

    And the number of samples we end up doing would depend on how well the campaign does. If it does better than expected, we can get over 100+ people recorded, and shoot for 150+. If it does poorly, we can just do 50 or 30. That way the study is flexible and won’t bomb either way.

    Thoughts?

    What are your thoughts on this?

    Would you change any parts of the experiment above?

    Thanks!

    • This topic was modified 1 week, 3 days ago by Auburn.
    • This topic was modified 1 week, 2 days ago by Auburn.
    Supah Protist
    Participant
    • Type: SeTi
    • Development: ll-l
    • F Attitude: Directive

    I think a big thing to keep in mind is quantification. I’m not sure that measuring people’s opinions on which signals someone is expressing qualifies as objective data. Setting up a protocol that only records the the quantitative correlates of the signals may be more reliable. For example, actually measuring the number of discrete shoulder shrug movements and comparing this to the number of discrete head nods as well as the times at which they occur. This would require forming perhaps quantitative but definitely discrete definitions of shoulder shrugs and head nods in this case. I don’t think setting up a dichotomous signal library is necessarily the best idea because it would rely on qualitative descriptions and interpretations as opposed to quantitative data that could speak for itself.

    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    Heya Supah!

    For example, actually measuring the number of discrete shoulder shrug movements and comparing this to the number of discrete head nods as well as the times at which they occur.

    Yes, have you seen the new multi-timestamp codifier with .CSV file export functionality? https://cognitivetype.com/forums/topic/multi-timestamp-support-for-codifier/

    My thinking is quite in agreement with yours, I think. The new codifier allows for individual signals to be tallied, at every second, so u can get a numerical count of how many of each signal was displayed, and work with that data via Excel/etc.

    I don’t think setting up a dichotomous signal library is necessarily the best idea because it would rely on qualitative descriptions and interpretations as opposed to quantitative data that could speak for itself.

    Well, the dichotomous setup would be necessary to see if what is being tested is a true dichotomy or a bell curve. And I think that’s a separate topic to the topic of qualitative-vs-quantitative.

    As for quantitative vs qualitative, there was some discussion of that in this post, I wonder if you saw it? https://cognitivetype.com/forums/topic/is-the-vultology-theory-somewhat-self-confirming/#post-19671 – In short, the idea would be that a new version of the code would be made (3.0) specifically designed to be fully quantitative.  Every signal would be registered as a motion vector, and be non-contingent on any other signal, so that correlations aren’t piggybacked onto each other artificially.

    This would allow for us to properly examine whether or not the signals currently grouped under (lets say) Fi are all statistically connected to each other, when structural contingency is removed by consolidating them into non-contingent signals. And also whether that group (if it exists as a discrete set) correlates to Te (as a discrete set). And whether Fi+Te is negatively correlated to Ti/Fe as a discrete set.

    I wonder what you think?

    • This reply was modified 1 week, 2 days ago by Auburn.
    • This reply was modified 1 week, 2 days ago by Auburn.
    Supah Protist
    Participant
    • Type: SeTi
    • Development: ll-l
    • F Attitude: Directive

    I think what I meant by “dichotomous signal” is referencing the interpretative nature of the reading that would be done. The dichotomy can be used to interpret the data, but I don’t think it should be used in gathering the data. Setting up a dichotomous signal library makes people choose between two options, which would mean that they may bias the aggregate results. If people are looking for Te 1 vs. Fe 1 they may potentially be biased towards seeing Te in general or Fe in general if they already know what they are looking for.

    A key issue is that there aren’t quantitative definitions of the signals at present. If you were to use the current qualitative signal definitions, it would be hard to saying anything of value about the patterns in the “data” because the “data” would actually be subjective. Many languages have a word for red, people can identify red things reliably. However, red is not a discrete property of any object. It is an arbitrary subregion of a spectrum. What I’m saying is that there’s no science of red or color itself because they are subjective concepts as opposed to an objective ones. There’s no discrete transition that occurs between red and orange light so you can’t study the difference objectively.

    Saying “we all consistently identify this apple as red” doesn’t mean much if the terms haven’t been defined quantitatively. What do you mean by apple? What do you mean by red? Better statements may be “this apple weighs x ounces” or “this apple contains carbohydrates”. This is because these statements can be empirically validated through quantitative methods. If you use subjective definitions for the signals then the relations between the signals won’t really mean anything.   I guess the main point is that what people think something is doesn’t count as objective scientific data since it’s not a quantitative measurement. I hope I’m being clear enough.

    Also, the survey will be important since none of these vultological correlations matter if they don’t correlate with behavioral and psychological variables.

    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    Oh, I don’t think the goal of this idea is to be that reductive.

    Granted, I don’t think you’re wrong, but the level of specificity your proposal would require is astronomical. For example, a whole study would have to be dedicated to identifying when and how something counts as a head nod or shoulder shrug [or anything else]. How much lift of the shoulder counts as a positive mark, or how much accentuation of the chin counts as a nod? With what velocity, and from what angle? Is it measured in inches? Inches per second? Does one inch not cross the threshold, but three does? Or is it proportional to the person’s body size? Etc.

    (And then there’s the infinite variability in human mannerisms overlapping. For example, what if they’re leaning their arm on the chair arm-wrest, which naturally lifts their shoulder 2 inches. Do we count this as a shoulder shrug, or is it only in relation to a certain baseline from which it rises and falls quickly?)

    By this sort of logic, cataloguing human movement into discrete signals, objectively, either cannot be done, or it would take training a neural network to identify clustering (as I mentioned) — so long as we accept the results of a neural network itself. But then that’s contingent on the training data, which can vary. So we’d have to decide at some point how much training data is good-enough, what data is good enough, and the list goes on.

    I’ve been keeping up with the progress on machine learning, but I haven’t seen anything that has catalogued natural human movement accurately, yet. And this is with well funded labs. So there’s no chance for CT to do something like this anytime soon. Therefore, we have to make concessions.

    Concessions

    What I’m proposing is that we allow for some general assumptions, but have those assumptions be informed by quantitative descriptions.

    So for example, lets say that we define a Je-5 Projecting Hands gesture as a forward protrusion of the arm where the elbow is in front of the torso area, the wrist is in front of the elbow, and the elbow angle is no less than 90 degrees wide. We have this as a verbal description but we don’t actually capture the sample from multiple camera angles and go measure the degrees in the elbow angle at each frame. We use human eyes to approximate these values.

    I find this level of granularity in describing signals, as motion vectors, is acceptable for our purposes at the moment, no? Once again, the aim wouldn’t be to get scientifically published (precisely for reasons that you mentioned), but to perform a pilot study that can make a few concessions on the nitty gritty, while getting an overall sense of whether something is there or not.

    What do you think of something like this, or do you think that any concessions of this degree would make the data useless?

    • This reply was modified 1 week, 2 days ago by Auburn.
    • This reply was modified 1 week, 2 days ago by Auburn.
    • This reply was modified 1 week, 2 days ago by Auburn.
    Supah Protist
    Participant
    • Type: SeTi
    • Development: ll-l
    • F Attitude: Directive

    I think I may need to contextualize this discussion.

    Experiments test falsifiable hypotheses.

    What falsifiable hypothesis do you want to test?

    If there are several, feel free to list them.

    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    Well, chiefly the ones listed above as:

    • If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency (and moving to code 3.0).
    • If, for the most part, the function axes are a bimodal distribution, under controlled settings, or not.
    • If so/not, how much signal mixing there is.
    • If there is any correlation between the psychological survey results and the vultology code results.

    But other things could also be evaluated.

    • This reply was modified 1 week, 2 days ago by Auburn.
    fayest42
    Participant
    • Type: FiNe
    • Development: ll--
    • F Attitude: Unseelie

    I wonder if an artificial neural network could be helpful? I don’t know much about them, but perhaps an ANN could learn to type people if we just fed it a bunch of videos of people that we’ve typed. Then we could give it different videos of other people we’ve typed to see if it categorizes them the same way we do. If the ANN consistently assigned people the same type that we do, then we would know that there are objective signals. Plus then we’d have a have a way of typing people that didn’t rely on humans, which would be quite useful in a variety of different ways. I’m not sure how ambitious something like that is, but it doesn’t seem totally outside the realm of possibility.

    Edit: Oh, I didn’t see some of the posts above before I posted this. Looks like you’ve already had this thought 🙂

    • This reply was modified 1 week, 2 days ago by fayest42.
    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    @fayest42 It’s a great thought though!

    And yes, precisely. The practical applications would be enormous. It would have to be trained indeed. I was talking to Staas about how, if we refine the codex, then generate some 100 CSV’s of 100 videos, as the starting data, we might get a level of predictability. But yea, I don’t know how involved this would be either.

    Spoiler:

    I came across this today:

    It’s used for sign language recognition, but the programming structure might be similar enough to be repurposed. Not sure. The guy put the code up in GitHub.

    • This reply was modified 1 week, 2 days ago by Auburn.
    fayest42
    Participant
    • Type: FiNe
    • Development: ll--
    • F Attitude: Unseelie

    @Auburn That sign language translator looks really neat! I don’t know enough about programming to know whether the code might be useful to us, but perhaps.

    if we refine the codex, then generate some 100 CSV’s of 100 videos, as the starting data, we might get a level of predictability.

    Forgive me if I’m wrong about how ANN’s work, but isn’t it true that you wouldn’t need to actually teach them specific signals through the csv’s? Wouldn’t you be able to just give it a bunch of videos and tell it what type each person is and then the ANN would figure out for itself how to recognize different types?

    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    Oh I don’t know too much about it either. I only just started diving into them. I think there are various methods and one is called ‘supervised‘ learning, where an initial training set is utilized. Neural networks can be trained at different levels of supervision, I think. Seems different approaches work for different use cases.

    If the only feedback the neural network got was whether it got the type right or wrong, that might not be the best feedback for it to learn. Or it could be, but generally the less supervision there is, the greater the sample size has to be. I’m not sure if we’d need thousands or tens of thousands of videos to train it. But each video is thousands of still images, so I have no idea how intensive that would be.

    Another thing to consider is… most work on AI networks and visual recognition that I’ve seen deal with properly labeling static photos, which is hard enough just on its own, without stacking anything else on top of it. So for a neural network to even recognize that there’s a person in the video, is its own challenge. So is anything nuanced like if they’re wearing a certain shirt, and at certain frames the shirt shape or color makes it ambiguous to tell it apart from the arms/etc.

    We’d run into a lot of other basic visual recognition problems before we can even test out the CT hypothesis. So I think a more targeted optimization would be needed for this, to help simplify the matter. Something like full-body skeleton tracking:

    Spoiler:

    If something like a Kinect (which needs no physical markers) can be used to track the body, then the raw data for the joints can be extracted. Once the raw data is there, we can run just those values through the neural network like you described, giving it minimal supervision. I just think that having it solve the computer vision problem at the same time as everything else is too much.

    But then, if Kinect is not accurate enough, this might mean that our samples would need to wear body trackers!

    Spoiler:

    Haha, that’d be a fun surprise for our 100 strangers, eh? “Here, put this on!”

    • This reply was modified 1 week, 2 days ago by Auburn.
    • This reply was modified 1 week, 2 days ago by Auburn.
    • This reply was modified 1 week, 2 days ago by Auburn.
    fayest42
    Participant
    • Type: FiNe
    • Development: ll--
    • F Attitude: Unseelie

    Neural networks can be trained at different levels of supervision, I think.

    Ah, that makes sense. In that case, some amount of supervision could be useful. But I’d also be curious to see if it could figure out signals that we haven’t noticed.

    Re: kinect, yes, that makes a lot of sense. I imagine it would be much faster and would require fewer samples for it to learn just from raw movement data rather than from videos. Heh, body trackers could be fun 🙂

    Xirailuyo
    Participant
    • Type: NeTi
    • Development: l-l-
    • F Attitude: Adaptive

    Okay, so I wrote a long post about this as I work in deep learning. Yet, I’m editing this after thinking about the subject for a few minutes. CT does not have the data for deep learning. You need probably 100 examples of each vultological signal in a short clip (1-3 seconds each?). With enough work, you could have the algorithm trace the right muscles and parts of the body. But without a whole lot of samples of each signal, you’re not going to be able to get an algorithm to learn the appropriate quantitative measures. It’s the best method to measure signals using an algorithm, but after some more consideration, I really don’t think the data problem is manageable with the current team.

    Your best bet is to have several people independently quantify each signal, then statistically measure how much their evaluations correlate. For instance, two people select Fi-2 within 3 seconds of the same time. That’s a sign you’re actually measuring the same thing.

    I edited out most of the unimportant stuff from the original post, as it was mostly written humorously or as an offer to work on a project that I’m not sure is feasible. The remaining information is below.

    A completely novel theory of human cognition is not going to fly for scientific papers. You’re going to need to pick out something simple, such as the clustering of certain facial features. Then in another paper, you apply a cognitive assessment/test showing that there’s a correlation between these facial clusters and psychological phenomena. I’m sure you have already considered this.

    For the eventual psychological assessment, whenever it is utilized and/or published, you want to closely follow the systems used by other personality assessments, such as FFM (without using FFM itself of course). While the tests obviously have to be tailored to CT, the more you can use already tested material, the better off you are. This is of course, if your goal is for CT to be taken seriously in scientific literature. Obviously, you can just develop your own questions based on your own thoughts on accuracy, and quite possibly get better results. You just might not get published.

    Funding for participants is generally acceptable, though there are standards for how much should be compensated. There are also methods for how to select a more or less random population, though most studies do primarily utilize college students or other limited groups.

    Let me know if you want to discuss this more over voice chat, or if anyone has any questions in particular. I’m extremely familiar with deep learning and psychology.

    • This reply was modified 1 week, 2 days ago by Xirailuyo.
    Auburn
    Keymaster
    • Type: TiNe
    • Development: l--l
    • F Attitude: Adaptive

    hey Xirailuyo! Thanks for dropping by. 🙂

    Getting 100 clips per signal is a hell of a grind, but doable if we wanted to go that route. We kinda need that anyway.

    A completely novel theory of human cognition is not going to fly for scientific papers.

    Naturally.

    I’ve been aware of this from the very start, which is part of why CT took this independent research path– because the development of the model requires associative and holistic (non-reductive) thinking for it to be conceptualized (and refined) as a whole during the early stages. Much like a neural network learning how to ‘see’, CT started off by a group of us having a ‘fuzzy’ view of the entirety of human motions/mannerisms, in order to properly orient itself across the aggregated thresholds dividing all its phenomenon. We thus acted a bit like this neural network and fed ourselves thousands of videos to extract patterns. Things like Je, Ji, Pe, Pi manifest as clusters at this level of holistic/panoramic resolution, as strong natural grupings of signals.

    Then, verifying or falsifying these conclusions would each take a whole independent study. But the idea is that this would be repeatable, and that other neural networks that examine human motions would also cluster human energetics first around Je/Ji/Pe/Pi, or something very close to it.

    I think this can be tested, through deep learning + body tackers, if I’m not mistaken Xirailuyo? Could you think of an experiment where a neural network could be fed body-tracked data of people sitting on a chair being interviewed, and see if the motion vectors fall into discrete categories?

    What do you think of something like this, as one possible test of the energetics?

    Sander
    Participant
    • Type: NeFi
    • Development: lll-
    • F Attitude: Seelie
    @auburn wrote:

    • If, for the most part, the function axes are a bimodal distribution under controlled settings. (or not)
    • How much signal mixing there really is.
    • If the items grouped under each function/axes are statistically correlated to each other, after fixing the issues of signal redundancy and automatic dependency.
    • If there is any correlation between the psychological survey results and the vultology code results.

    You already know the statistical significance of your first and last hypothesis. The recent quizzes already showed statistical significance, and the next quizzes can verify further improvements. And the cybernetic Big Five statistics already show the psychological relevance of clusters that were intended to map on function-attitudes.

    Moreover, you expect energetics to be primary; so, then bimodality shows up more when functions are developed. However, most test subjects are expected to have only one out of four functions developed; so, the average separation of function-axis isn’t a great measure for differentiating CT personalities.

    Hence, I suggest testing the energetics first; apart from greatly reducing complexity, that also makes good PR material for crowdfunding a follow-up experiment.

Viewing 15 posts - 1 through 15 (of 17 total)
  • You must be logged in to reply to this topic.
© Copyright 2012-2020 | CognitiveType.com
This website's articles, its reading methodology and practices are the intellectual property of J.E. Sandoval.
Animaged GIFs, images and videos belong to their respective owners.